Jul 24 13:32:05.213550 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 13:32:05.213563 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 13:32:05.213573 ip-10-0-129-199 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 13:32:05.213935 ip-10-0-129-199 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 13:32:15.378450 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 13:32:15.378471 ip-10-0-129-199 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c0d57e07e9694212a45c79f12057cf84 --
Jul 24 13:33:55.837752 ip-10-0-129-199 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 13:33:56.243642 ip-10-0-129-199 kubenswrapper[2527]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:33:56.243642 ip-10-0-129-199 kubenswrapper[2527]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 13:33:56.243642 ip-10-0-129-199 kubenswrapper[2527]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:33:56.243642 ip-10-0-129-199 kubenswrapper[2527]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 13:33:56.243642 ip-10-0-129-199 kubenswrapper[2527]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:33:56.244725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.244601    2527 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 13:33:56.249009 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.248989    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.249009 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249009    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.249009 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249013    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249016    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249019    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249022    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249025    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249029    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249032    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249035    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249038    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249041    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249044    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249046    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249049    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249052    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249055    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249057    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249060    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249063    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249067    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.249112 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249070    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249074    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249076    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249083    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249085    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249088    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249091    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249093    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249096    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249099    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249101    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249104    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249107    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249109    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249113    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249116    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249119    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249121    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249124    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249127    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.249588 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249129    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249132    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249135    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249137    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249140    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249143    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249146    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249148    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249151    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249153    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249155    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249158    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249161    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249165    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249168    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249170    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249173    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249176    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249178    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.250074 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249181    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249184    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249188    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249192    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249196    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249198    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249201    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249204    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249209    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249212    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249214    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249217    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249219    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249221    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249224    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249227    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249229    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249234    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249237    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.250550 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249239    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249242    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249244    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249247    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249249    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249252    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249255    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249257    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249260    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249263    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249265    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249267    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249270    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249272    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249275    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249277    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249280    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249282    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249285    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.249287    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.251064 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250023    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250030    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250033    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250036    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250039    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250042    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250044    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250047    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250049    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250052    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250055    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250058    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250061    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250063    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250066    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250068    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250071    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250074    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250077    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250080    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.251584 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250082    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250085    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250087    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250090    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250092    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250095    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250097    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250100    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250102    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250105    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250107    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250109    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250112    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250115    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250117    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250120    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250123    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250125    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250128    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250131    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.252077 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250133    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250137    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250139    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250142    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250147    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250150    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250153    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250156    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250159    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250161    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250165    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250168    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250171    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250174    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250177    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250179    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250182    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250184    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250187    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.252599 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250189    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250192    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250195    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250197    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250200    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250203    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250205    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250208    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250210    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250213    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250215    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250218    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250220    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250223    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250226    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250228    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250230    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250233    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250236    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250239    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.253078 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250241    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250244    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250246    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250248    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250251    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250253    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250255    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250258    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250262    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250264    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250268    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250271    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250274    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250277    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250279    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250282    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250285    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250288    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250290    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.253595 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250294    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250701    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250706    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250708    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250711    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250714    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250716    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250719    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250722    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250725    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250727    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250731    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250733    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250736    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250738    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250741    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250743    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250746    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250748    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250751    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.254055 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250753    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250757    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250761    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250763    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250768    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250771    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250774    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250776    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250779    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250781    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250784    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250786    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250789    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250791    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250794    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250796    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250799    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250802    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250806    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.254557 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250809    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250812    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250814    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250817    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250819    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250822    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250825    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250828    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250831    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250833    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250836    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250838    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250841    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250843    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250846    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250848    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250850    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250855    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250857    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250860    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.255051 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250863    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250866    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250868    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250871    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250873    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250876    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250878    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250881    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250883    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250885    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250888    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250890    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250894    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250896    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250899    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250901    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250904    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250906    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250909    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250911    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.255564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250914    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250916    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250919    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250921    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250923    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250926    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250928    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250932    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250935    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250939    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250942    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250944    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250947    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250949    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250952    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250954    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250957    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250960    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250962    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.256131 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250964    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.250967    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251043    2527 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251050    2527 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251056    2527 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251061    2527 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251065    2527 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251068    2527 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251072    2527 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251077    2527 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251080    2527 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251083    2527 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251087    2527 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251091    2527 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251094    2527 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251096    2527 flags.go:64] FLAG: --cgroup-root=""
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251099    2527 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251102    2527 flags.go:64] FLAG: --client-ca-file=""
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251105    2527 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251108    2527 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251112    2527 flags.go:64] FLAG: --cluster-domain=""
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251115    2527 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251119    2527 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251130    2527 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 13:33:56.256645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251134    2527 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251138    2527 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251141    2527 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251144    2527 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251147    2527 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251150    2527 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251153    2527 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251156    2527 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251159    2527 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251162    2527 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251166    2527 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251168    2527 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251171    2527 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251174    2527 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251177    2527 flags.go:64] FLAG: --enable-server="true"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251180    2527 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251184    2527 flags.go:64] FLAG: --event-burst="100"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251187    2527 flags.go:64] FLAG: --event-qps="50"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251189    2527 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251192    2527 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251195    2527 flags.go:64] FLAG: --eviction-hard=""
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251200    2527 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251203    2527 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251205    2527 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251208    2527 flags.go:64] FLAG: --eviction-soft=""
Jul 24 13:33:56.257217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251211    2527 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251214    2527 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251216    2527 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251219    2527 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251222    2527 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251225    2527 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251228    2527 flags.go:64] FLAG: --feature-gates=""
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251232    2527 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251235    2527 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251238    2527 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251241    2527 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251244    2527 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251247    2527 flags.go:64] FLAG: --help="false"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251250    2527 flags.go:64] FLAG: --hostname-override="ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251253    2527 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251255    2527 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251258    2527 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251261    2527 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251265    2527 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251267    2527 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251270    2527 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251273    2527 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251276    2527 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 13:33:56.257843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251278    2527 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251281    2527 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251284    2527 flags.go:64] FLAG: --kube-reserved=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251287    2527 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251289    2527 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251292    2527 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251295    2527 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251298    2527 flags.go:64] FLAG: --lock-file=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251301    2527 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251303    2527 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251306    2527 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251312    2527 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251314    2527 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251317    2527 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251320    2527 flags.go:64] FLAG: --logging-format="text"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251322    2527 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251325    2527 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251328    2527 flags.go:64] FLAG: --manifest-url=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251331    2527 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251336    2527 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251339    2527 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251343    2527 flags.go:64] FLAG: --max-pods="110"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251347    2527 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251349    2527 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251352    2527 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 13:33:56.258568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251355    2527 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251357    2527 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251360    2527 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251364    2527 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251372    2527 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251375    2527 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251378    2527 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251381    2527 flags.go:64] FLAG: --pod-cidr=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251384    2527 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251389    2527 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251392    2527 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251395    2527 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251398    2527 flags.go:64] FLAG: --port="10250"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251400    2527 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251404    2527 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5c71a7dd422e386"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251407    2527 flags.go:64] FLAG: --qos-reserved=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251410    2527 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251412    2527 flags.go:64] FLAG: --register-node="true"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251415    2527 flags.go:64] FLAG: --register-with-taints=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251419    2527 flags.go:64] FLAG: --registry-burst="10"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251421    2527 flags.go:64] FLAG: --registry-qps="5"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251424    2527 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251427    2527 flags.go:64] FLAG: --reserved-memory=""
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251444    2527 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251447    2527 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 13:33:56.259321 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251450    2527 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251452    2527 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251455    2527 flags.go:64] FLAG: --runonce="false"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251458    2527 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251460    2527 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251463    2527 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251466    2527 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251469    2527 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251472    2527 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251475    2527 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251479    2527 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251482    2527 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251485    2527 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251487    2527 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251490    2527 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251493    2527 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251496    2527 flags.go:64] FLAG: --system-cgroups=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251499    2527 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251504    2527 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251507    2527 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251509    2527 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251513    2527 flags.go:64] FLAG: --tls-min-version=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251517    2527 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251519    2527 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251522    2527 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 13:33:56.260094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251525    2527 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251528    2527 flags.go:64] FLAG: --v="2"
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251532    2527 flags.go:64] FLAG: --version="false"
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251537    2527 flags.go:64] FLAG: --vmodule=""
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251541    2527 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.251544    2527 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251665    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251669    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251673    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251676    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251680    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251683    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251685    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251688    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251691    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251693    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251696    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251700    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251702    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251705    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251707    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.260810 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251710    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251713    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251715    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251717    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251720    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251722    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251725    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251727    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251730    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251734    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251738    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251741    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251744    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251747    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251751    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251753    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251756    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251759    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251761    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.261410 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251764    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251767    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251771    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251774    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251777    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251779    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251782    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251785    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251787    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251791    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251793    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251795    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251798    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251800    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251803    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251805    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251808    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251810    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251813    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.261996 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251815    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251818    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251820    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251822    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251832    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251835    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251837    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251840    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251845    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251848    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251851    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251853    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251856    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251858    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251861    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251864    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251867    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251869    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251871    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251874    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.262533 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251876    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251878    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251886    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251889    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251891    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251894    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251897    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251899    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251902    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251905    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251907    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251910    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251912    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251915    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251917    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251920    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251923    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251926    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251928    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251931    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.263033 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251934    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251937    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251940    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251942    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251944    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.251947    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.252810    2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.259993    2527 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260013    2527 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260034    2527 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260044    2527 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260106    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260112    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260115    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260118    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.263564 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260121    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260124    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260127    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260130    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260133    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260136    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260138    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260141    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260144    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260146    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260148    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260151    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260153    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260155    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260158    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260160    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260163    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260165    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260168    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260171    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.263947 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260173    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260175    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260178    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260180    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260184    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260195    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260198    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260201    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260204    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260207    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260209    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260212    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260214    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260217    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260219    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260221    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260224    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260226    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260229    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260231    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.264440 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260233    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260236    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260238    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260241    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260243    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260246    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260248    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260251    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260253    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260257    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260260    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260263    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260266    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260268    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260271    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260273    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260275    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260278    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260287    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.264929 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260290    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260293    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260295    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260297    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260299    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260303    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260307    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260309    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260312    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260315    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260318    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260320    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260323    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260325    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260328    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260330    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260333    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260335    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260337    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.265396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260340    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260342    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260345    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260347    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260350    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260352    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260354    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260357    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260359    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260362    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260364    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260367    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260370    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260380    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260382    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260385    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.265887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260387    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260393    2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260564    2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260569    2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260572    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260575    2527 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260578    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260581    2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260583    2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260586    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260589    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260592    2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260595    2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260597    2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260600    2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:33:56.266279 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260603    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260605    2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260608    2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260610    2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260613    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260616    2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260618    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260621    2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260623    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260625    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260628    2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260630    2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260632    2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260635    2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260638    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260646    2527 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260649    2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260651    2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260654    2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260656    2527 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:33:56.266661 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260658    2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260661    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260663    2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260666    2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260668    2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260670    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260673    2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260675    2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260677    2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260680    2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260683    2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260685    2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260688    2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260691    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260693    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260695    2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260698    2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260700    2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260702    2527 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260705    2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:33:56.267138 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260707    2527 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260709    2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260711    2527 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260714    2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260716    2527 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260719    2527 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260722    2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260724    2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260733    2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260736    2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260738    2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260740    2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260743    2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260746    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260748    2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260750    2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260754    2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260758    2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260761    2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260763    2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:33:56.267641 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260765    2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260768    2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260770    2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260772    2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260774    2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260777    2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260779    2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260781    2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260783    2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260786    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260788    2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260790    2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260793    2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260796    2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260799    2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260804    2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260807    2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260810    2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260813    2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:33:56.268150 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260816    2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260818    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260827    2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260829    2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260832    2527 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260834    2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:56.260837    2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260842    2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.260994    2527 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.263174    2527 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.264663    2527 server.go:1019] "Starting client certificate rotation"
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.264773    2527 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.264817    2527 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:33:56.268629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.268015    2527 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:33:56.272640 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.272615    2527 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:33:56.281292 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.281271    2527 log.go:25] "Validated CRI v1 runtime API"
Jul 24 13:33:56.283483 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.283468    2527 log.go:25] "Validated CRI v1 image API"
Jul 24 13:33:56.284329 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.284310    2527 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 13:33:56.291201 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.291164    2527 fs.go:135] Filesystem UUIDs: map[02d67540-36ab-4946-9e33-a5d3c6d8c0cc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bdb599fd-c20b-4de6-8aad-9fc838d309b4:/dev/nvme0n1p4]
Jul 24 13:33:56.291283 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.291199    2527 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 13:33:56.296140 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.296118    2527 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:33:56.300572 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.300456    2527 manager.go:216] Machine: {Timestamp:2026-07-24 13:33:56.298621796 +0000 UTC m=+0.340669672 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098875 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b8673a104e39ff731e65760c7f8c SystemUUID:ec21b867-3a10-4e39-ff73-1e65760c7f8c BootID:c0d57e07-e969-4212-a45c-79f12057cf84 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cc:2c:c5:0a:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cc:2c:c5:0a:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:b2:d1:1c:7b:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 13:33:56.300572 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.300566    2527 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 13:33:56.300703 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.300657    2527 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 13:33:56.303755 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.303731    2527 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 13:33:56.303909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.303758    2527 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 13:33:56.305093 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.305081    2527 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 13:33:56.305131 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.305096    2527 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 13:33:56.305131 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.305103    2527 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:33:56.305851 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.305840    2527 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:33:56.306687 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.306677    2527 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 13:33:56.308621 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308609    2527 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:33:56.308793 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308783    2527 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 13:33:56.308835 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308825    2527 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 13:33:56.308868 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308845    2527 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 13:33:56.308868 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308856    2527 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 13:33:56.308868 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308867    2527 kubelet.go:398] "Adding apiserver pod source"
Jul 24 13:33:56.308970 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.308886    2527 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 13:33:56.309401 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.309387    2527 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:33:56.309480 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.309411    2527 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:33:56.313546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.313528    2527 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 13:33:56.315812 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.315795    2527 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 13:33:56.315906 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.315825    2527 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 13:33:56.317358 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317341    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317363    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317372    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317380    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317393    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317402    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317413    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317421    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317448    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317456    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317469    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 13:33:56.317474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.317478    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 13:33:56.318310 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.318297    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 13:33:56.318369 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.318312    2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 13:33:56.318718 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.318693    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 13:33:56.318802 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.318729    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 13:33:56.321577 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.321561    2527 server.go:1268] "Started kubelet"
Jul 24 13:33:56.321696 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.321640    2527 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 13:33:56.321743 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.321702    2527 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 13:33:56.322069 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.321724    2527 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 13:33:56.325914 ip-10-0-129-199 systemd[1]: Started Kubernetes Kubelet.
Jul 24 13:33:56.326596 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.326368    2527 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 13:33:56.327560 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.327534    2527 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 13:33:56.327642 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.327615    2527 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 13:33:56.333157 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.333108    2527 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 13:33:56.333878 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.333857    2527 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 13:33:56.333878 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.333867    2527 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 13:33:56.334014 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.333985    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.334411 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.333095    2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-199.ec2.internal.18c53d205086d462  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-199.ec2.internal,UID:ip-10-0-129-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-199.ec2.internal,},FirstTimestamp:2026-07-24 13:33:56.321530978 +0000 UTC m=+0.363578854,LastTimestamp:2026-07-24 13:33:56.321530978 +0000 UTC m=+0.363578854,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-199.ec2.internal,}"
Jul 24 13:33:56.334545 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.334472    2527 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 13:33:56.334545 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.334489    2527 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 13:33:56.334545 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.334505    2527 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 13:33:56.334674 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.334596    2527 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 13:33:56.334674 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.334605    2527 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 13:33:56.336891 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.336805    2527 factory.go:55] Registering systemd factory
Jul 24 13:33:56.336998 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.336896    2527 factory.go:223] Registration of the systemd container factory successfully
Jul 24 13:33:56.339220 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.339201    2527 factory.go:153] Registering CRI-O factory
Jul 24 13:33:56.339220 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.339222    2527 factory.go:223] Registration of the crio container factory successfully
Jul 24 13:33:56.339335 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.339277    2527 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 13:33:56.339335 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.339305    2527 factory.go:103] Registering Raw factory
Jul 24 13:33:56.339335 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.339322    2527 manager.go:1196] Started watching for new ooms in manager
Jul 24 13:33:56.339586 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.339557    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 13:33:56.339586 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.339575    2527 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 13:33:56.340015 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.340003    2527 manager.go:349] Starting recovery of all containers
Jul 24 13:33:56.348704 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.348687    2527 manager.go:354] Recovery completed
Jul 24 13:33:56.350046 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.350025    2527 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnc7l"
Jul 24 13:33:56.352856 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.352843    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.354725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.354708    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.354805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.354740    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.354805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.354754    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.355304 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.355290    2527 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 13:33:56.355304 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.355302    2527 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 13:33:56.355393 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.355318    2527 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:33:56.355393 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.355353    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.356014 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.355997    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.356055 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.356027    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.356055 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.356041    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.356960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.356944    2527 policy_none.go:49] "None policy: Start"
Jul 24 13:33:56.357030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.356965    2527 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 13:33:56.357030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.356977    2527 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 13:33:56.357686 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.357210    2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-199.ec2.internal.18c53d20528155a9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-199.ec2.internal,UID:ip-10-0-129-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-199.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-199.ec2.internal,},FirstTimestamp:2026-07-24 13:33:56.354725289 +0000 UTC m=+0.396773168,LastTimestamp:2026-07-24 13:33:56.354725289 +0000 UTC m=+0.396773168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-199.ec2.internal,}"
Jul 24 13:33:56.357971 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.357961    2527 policy_none.go:47] "Start"
Jul 24 13:33:56.359136 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.358986    2527 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnc7l"
Jul 24 13:33:56.402145 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402122    2527 manager.go:337] "Starting Device Plugin manager"
Jul 24 13:33:56.402266 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.402164    2527 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 13:33:56.402266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402174    2527 server.go:87] "Starting device plugin registration server"
Jul 24 13:33:56.402523 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402511    2527 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 13:33:56.402574 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402528    2527 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 13:33:56.402635 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402617    2527 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 13:33:56.402719 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402706    2527 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 13:33:56.402719 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.402719    2527 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 13:33:56.403325 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.403302    2527 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 13:33:56.403405 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.403361    2527 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.418368 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.418341    2527 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2561 into the system container \"/system.slice\": <nil>"
Jul 24 13:33:56.503015 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.502884    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.504059 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.504022    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.504059 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.504063    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.504239 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.504077    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.504239 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.504104    2527 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.505702 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.505669    2527 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 13:33:56.507091 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.507070    2527 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 13:33:56.507192 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.507108    2527 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 13:33:56.507192 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.507138    2527 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 13:33:56.507192 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.507178    2527 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 13:33:56.509746 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.509709    2527 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:33:56.514485 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.514469    2527 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.514560 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.514492    2527 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-199.ec2.internal\": node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.553257 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.553227    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.608287 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.608254    2527 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"]
Jul 24 13:33:56.608394 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.608337    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.609305 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.609287    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.609409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.609317    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.609409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.609328    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.610741 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.610728    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.610874 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.610854    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.610909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.610892    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.611592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611577    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.611663 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611605    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.611663 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611614    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.611663 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611578    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.611750 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611689    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.611750 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.611707    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.613064 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.613050    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.613123 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.613078    2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:33:56.613823 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.613805    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:33:56.613897 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.613837    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:33:56.613897 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.613850    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:33:56.636085 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.636050    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.636085 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.636085    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.636260 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.636108    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f9f9e23f5a0fe49a38ad742dd8ed6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"2f9f9e23f5a0fe49a38ad742dd8ed6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.640348 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.640325    2527 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-199.ec2.internal\" not found" node="ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.644954 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.644926    2527 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-199.ec2.internal\" not found" node="ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.653318 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.653293    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.736617 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736577    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.736617 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736616    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.736847 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736639    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f9f9e23f5a0fe49a38ad742dd8ed6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"2f9f9e23f5a0fe49a38ad742dd8ed6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.736847 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736680    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f9f9e23f5a0fe49a38ad742dd8ed6c0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"2f9f9e23f5a0fe49a38ad742dd8ed6c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.736847 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736690    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.736847 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.736680    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d55cc1510d7fd2078798bc611e3e5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"44d55cc1510d7fd2078798bc611e3e5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.754238 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.754172    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.854745 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.854710    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:56.943262 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.943223    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.948161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:56.948143    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:56.955413 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:56.955391    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:57.056029 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.055939    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:57.156499 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.156463    2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found"
Jul 24 13:33:57.166969 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.166937    2527 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:33:57.234623 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.234590    2527 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:57.257950 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.257927    2527 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:33:57.260246 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.260231    2527 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:57.264072 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.264055    2527 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 13:33:57.264219 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.264198    2527 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 13:33:57.264260 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.264240    2527 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 13:33:57.264293 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.264257    2527 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a0127f4cd09464b9eb60cb7a54016ff7-5af16f43cfd343b1.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.129.199:32812->100.60.232.236:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:33:57.309485 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.309382    2527 apiserver.go:52] "Watching apiserver"
Jul 24 13:33:57.324305 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.324278    2527 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 13:33:57.326635 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.326602    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9nhlx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","openshift-multus/multus-additional-cni-plugins-szkt4","openshift-multus/network-metrics-daemon-gdskt","openshift-network-diagnostics/network-check-target-rj5bn","kube-system/konnectivity-agent-qnqt6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j","openshift-multus/multus-xqs49","openshift-network-operator/iptables-alerter-hbxwj","openshift-ovn-kubernetes/ovnkube-node-p8n58","kube-system/global-pull-secret-syncer-qq8rj","openshift-cluster-node-tuning-operator/tuned-ghlrl","openshift-dns/node-resolver-z26h4"]
Jul 24 13:33:57.328223 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.328203    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.329541 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.329521    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.330928 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.330906    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.331028 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.330989    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:33:57.331900 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.331884    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.332785 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.332769    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5gvh6\""
Jul 24 13:33:57.333166 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333149    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 13:33:57.333460 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333447    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333459    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333504    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333525    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333576    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333633    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333642    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 13:33:57.333691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333645    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-27l9r\""
Jul 24 13:33:57.334003 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333739    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:57.334003 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.333789    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:33:57.334003 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.333912    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 13:33:57.334105 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.334036    2527 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 13:33:57.334276 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.334263    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.334472 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.334458    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 13:33:57.334846 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.334831    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.336176 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.336104    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.336176 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.336148    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5vfnj\""
Jul 24 13:33:57.336593 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.336578    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zwwtv\""
Jul 24 13:33:57.336672 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.336641    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 13:33:57.336788 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.336774    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 13:33:57.337334 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.337319    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.337402 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.337370    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 13:33:57.337504 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.337488    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.337582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.337569    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.338788 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.338775    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.339836 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.339819    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 13:33:57.339930 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.339858    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.339930 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.339895    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 13:33:57.339930 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.339900    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lvsn9\""
Jul 24 13:33:57.339930 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.339858    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.340400 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340388    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.340484 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340466    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cnibin\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.340541 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.340466    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:33:57.340541 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340509    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qh6zf\""
Jul 24 13:33:57.340541 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340522    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.340634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340548    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-binary-copy\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.340634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340574    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340618    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/961928b2-f95d-4775-88c0-c80ab72bd7be-konnectivity-ca\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.340634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340628    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xpt82\""
Jul 24 13:33:57.340773 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340648    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-etc-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340773 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340673    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-bin\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340773 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340696    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q8q\" (UniqueName: \"kubernetes.io/projected/b5be829c-ee49-4416-9f38-a6d3cd79c176-kube-api-access-w2q8q\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340773 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340720    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8705eba6-6f23-4ad9-8a91-5cdee758f61a-host\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.340773 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340751    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-slash\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340774    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-var-lib-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340816    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340843    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovn-node-metrics-cert\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340867    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9xg\" (UniqueName: \"kubernetes.io/projected/8b318dd6-b05b-4f18-abc7-6af9858c0690-kube-api-access-4c9xg\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340882    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8705eba6-6f23-4ad9-8a91-5cdee758f61a-serviceca\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340898    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340915    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-systemd-units\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340928    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-ovn\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.340944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340942    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxjs\" (UniqueName: \"kubernetes.io/projected/78698a3e-c8e7-473e-87e2-1fb4894d2c59-kube-api-access-4nxjs\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340956    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/961928b2-f95d-4775-88c0-c80ab72bd7be-agent-certs\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340972    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-script-lib\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.340986    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-netns\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341011    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-netd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341039    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-config\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341076    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341100    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341122    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-os-release\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341143    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341162    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-systemd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341175    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-log-socket\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341190    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-env-overrides\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341214    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr94p\" (UniqueName: \"kubernetes.io/projected/8705eba6-6f23-4ad9-8a91-5cdee758f61a-kube-api-access-xr94p\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341227    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-node-log\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341241    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341255    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-kubelet\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.341817 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341271    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-system-cni-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.341817 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341593    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.341817 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341693    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 13:33:57.341817 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.341717    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.342276 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.342263    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.343986 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.343969    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cqxlw\""
Jul 24 13:33:57.344049 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.344009    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.344118 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.344104    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.344472 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.344455    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.346101 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.346082    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fb8xr\""
Jul 24 13:33:57.346101 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.346099    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 13:33:57.346240 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.346197    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 13:33:57.346368 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.346353    2527 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:33:57.363026 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.362968    2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 13:28:56 +0000 UTC" deadline="2028-04-05 11:06:46.275436288 +0000 UTC"
Jul 24 13:33:57.363026 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.363012    2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14901h32m48.912428344s"
Jul 24 13:33:57.403141 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.403111    2527 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbtwb"
Jul 24 13:33:57.415566 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.415545    2527 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbtwb"
Jul 24 13:33:57.435909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.435887    2527 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 13:33:57.442160 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442135    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/961928b2-f95d-4775-88c0-c80ab72bd7be-konnectivity-ca\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442168    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-etc-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442186    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-bin\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442203    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-hosts-file\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442221    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-slash\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442243    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-var-lib-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442261 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442256    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-etc-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442265    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-bin\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442264    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442292    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442303    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-slash\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442314    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovn-node-metrics-cert\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442332    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-var-lib-openvswitch\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442343    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8705eba6-6f23-4ad9-8a91-5cdee758f61a-serviceca\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442393    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-registration-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442423    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-os-release\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442570    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-multus-certs\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442601    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442629    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-ovn\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442653    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/961928b2-f95d-4775-88c0-c80ab72bd7be-agent-certs\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442677    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-script-lib\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442700    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-conf-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442714    2527 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442725    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbd8k\" (UniqueName: \"kubernetes.io/projected/99cc38f8-6448-4047-a377-018ed5c08a5f-kube-api-access-mbd8k\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.442737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442745    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8705eba6-6f23-4ad9-8a91-5cdee758f61a-serviceca\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442749    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysconfig\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442825    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-ovn\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442828    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-run\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442859    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-netns\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442867    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/961928b2-f95d-4775-88c0-c80ab72bd7be-konnectivity-ca\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442893    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-netns\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442921    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-config\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.442961    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443065    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443106    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443141    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-systemd\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.443069    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443164    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knbd\" (UniqueName: \"kubernetes.io/projected/6d72c928-1cc5-4573-b71f-25c0fc2f0660-kube-api-access-7knbd\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443174    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443190    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4f5\" (UniqueName: \"kubernetes.io/projected/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-kube-api-access-dc4f5\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.443404 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.443226    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:33:57.943207001 +0000 UTC m=+1.985254869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443245    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443261    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-script-lib\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443277    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-systemd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443318    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-run-systemd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443338    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-log-socket\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443359    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-env-overrides\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443381    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-multus\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443383    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-log-socket\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443407    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-tuning-conf-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443417    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-lib-modules\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443466    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovnkube-config\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443483    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-tmp-dir\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443518    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-node-log\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443559    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hkb\" (UniqueName: \"kubernetes.io/projected/15520a7b-37bd-4094-9b6c-b542c254513d-kube-api-access-k6hkb\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443585    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-cni-binary-copy\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443599    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-node-log\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.444282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443607    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-var-lib-kubelet\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443660    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-system-cni-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443676    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cnibin\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443692    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443727    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-modprobe-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443737    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cnibin\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443744    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-kubernetes\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443736    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-system-cni-dir\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443835    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-dbus\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443862    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-binary-copy\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443907    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-device-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443940    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-system-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443984    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-cnibin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444005    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.443941    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5be829c-ee49-4416-9f38-a6d3cd79c176-env-overrides\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444019    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-sys\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444106    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-host\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444133    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q8q\" (UniqueName: \"kubernetes.io/projected/b5be829c-ee49-4416-9f38-a6d3cd79c176-kube-api-access-w2q8q\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444157    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8705eba6-6f23-4ad9-8a91-5cdee758f61a-host\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444214    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-conf\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444240    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9xg\" (UniqueName: \"kubernetes.io/projected/8b318dd6-b05b-4f18-abc7-6af9858c0690-kube-api-access-4c9xg\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444331    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-kubelet\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444335    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8705eba6-6f23-4ad9-8a91-5cdee758f61a-host\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444359    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-systemd-units\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444363    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444387    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxjs\" (UniqueName: \"kubernetes.io/projected/78698a3e-c8e7-473e-87e2-1fb4894d2c59-kube-api-access-4nxjs\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444414    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-tmp\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444422    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78698a3e-c8e7-473e-87e2-1fb4894d2c59-cni-binary-copy\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444478    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-systemd-units\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444519    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-netd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444546    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444563    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-cni-netd\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444579    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-k8s-cni-cncf-io\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444635    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d72c928-1cc5-4573-b71f-25c0fc2f0660-host-slash\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.445592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444663    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-os-release\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444701    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr94p\" (UniqueName: \"kubernetes.io/projected/8705eba6-6f23-4ad9-8a91-5cdee758f61a-kube-api-access-xr94p\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444723    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-sys-fs\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444745    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-netns\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444769    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9qn\" (UniqueName: \"kubernetes.io/projected/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-kube-api-access-lc9qn\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444800    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444822    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78698a3e-c8e7-473e-87e2-1fb4894d2c59-os-release\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444827    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-socket-dir-parent\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444865    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-bin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444889    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-etc-kubernetes\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444910    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-tuned\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444934    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-kubelet-config\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.444974    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445000    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445020    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-kubelet\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445034    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-socket-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.446132 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445052    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-daemon-config\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445087    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445108    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-hostroot\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445128    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-kubelet\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445131    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d72c928-1cc5-4573-b71f-25c0fc2f0660-iptables-alerter-script\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445155    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.445129    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5be829c-ee49-4416-9f38-a6d3cd79c176-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.446067    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5be829c-ee49-4416-9f38-a6d3cd79c176-ovn-node-metrics-cert\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.446722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.446198    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/961928b2-f95d-4775-88c0-c80ab72bd7be-agent-certs\") pod \"konnectivity-agent-qnqt6\" (UID: \"961928b2-f95d-4775-88c0-c80ab72bd7be\") " pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.455580 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.455555    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:33:57.455580 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.455578    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:33:57.455895 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.455590    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:57.455895 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.455699    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:33:57.955660655 +0000 UTC m=+1.997708520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:57.458003 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.457959    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxjs\" (UniqueName: \"kubernetes.io/projected/78698a3e-c8e7-473e-87e2-1fb4894d2c59-kube-api-access-4nxjs\") pod \"multus-additional-cni-plugins-szkt4\" (UID: \"78698a3e-c8e7-473e-87e2-1fb4894d2c59\") " pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.458133 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.458115    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9xg\" (UniqueName: \"kubernetes.io/projected/8b318dd6-b05b-4f18-abc7-6af9858c0690-kube-api-access-4c9xg\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.458929 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.458907    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr94p\" (UniqueName: \"kubernetes.io/projected/8705eba6-6f23-4ad9-8a91-5cdee758f61a-kube-api-access-xr94p\") pod \"node-ca-9nhlx\" (UID: \"8705eba6-6f23-4ad9-8a91-5cdee758f61a\") " pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.459094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.459078    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q8q\" (UniqueName: \"kubernetes.io/projected/b5be829c-ee49-4416-9f38-a6d3cd79c176-kube-api-access-w2q8q\") pod \"ovnkube-node-p8n58\" (UID: \"b5be829c-ee49-4416-9f38-a6d3cd79c176\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.546019 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.545988    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-conf-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.546161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546023    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbd8k\" (UniqueName: \"kubernetes.io/projected/99cc38f8-6448-4047-a377-018ed5c08a5f-kube-api-access-mbd8k\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.546161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546040    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysconfig\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546057    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-run\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546108    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysconfig\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546161 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546124    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-conf-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546194    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546217    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-run\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546222    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546239    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-systemd\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546247    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546255    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knbd\" (UniqueName: \"kubernetes.io/projected/6d72c928-1cc5-4573-b71f-25c0fc2f0660-kube-api-access-7knbd\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546280    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4f5\" (UniqueName: \"kubernetes.io/projected/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-kube-api-access-dc4f5\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546308    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-multus\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546310    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-systemd\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546311    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546332    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-lib-modules\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.546360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546340    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-multus\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546475    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-tmp-dir\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546518    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hkb\" (UniqueName: \"kubernetes.io/projected/15520a7b-37bd-4094-9b6c-b542c254513d-kube-api-access-k6hkb\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546530    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-lib-modules\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546546    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-cni-binary-copy\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546570    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-var-lib-kubelet\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546604    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-modprobe-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546627    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-kubernetes\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546648    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-dbus\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546678    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-var-lib-kubelet\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546683    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-device-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546738    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-device-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546735    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-system-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546757    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-kubernetes\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546782    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-system-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546787    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-cnibin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546807    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-tmp-dir\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546852    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-modprobe-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.547497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546811    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546896    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-sys\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546904    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-d\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546911    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-dbus\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546897    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-cnibin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546920    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-host\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546934    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-sys\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546957    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-host\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546960    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-conf\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.546990    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-kubelet\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547022    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-tmp\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547055    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-sysctl-conf\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547058    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-k8s-cni-cncf-io\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547067    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-kubelet\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547073    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-cni-binary-copy\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547082    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d72c928-1cc5-4573-b71f-25c0fc2f0660-host-slash\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547131    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d72c928-1cc5-4573-b71f-25c0fc2f0660-host-slash\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547116    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-k8s-cni-cncf-io\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.548327 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547162    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-sys-fs\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547202    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-netns\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547225    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9qn\" (UniqueName: \"kubernetes.io/projected/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-kube-api-access-lc9qn\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547248    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547267    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-sys-fs\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547274    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-netns\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547271    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-socket-dir-parent\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547323    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-cni-dir\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547324    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-socket-dir-parent\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547324    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-bin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547352    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-var-lib-cni-bin\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547367    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-etc-kubernetes\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547390    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-tuned\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547412    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-kubelet-config\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547420    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-etc-kubernetes\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547458    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547501    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-socket-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547518    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c72de-350d-4005-9d29-bf40ae47ac5c-kubelet-config\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:57.549002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547526    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-daemon-config\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547550    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-hostroot\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547575    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d72c928-1cc5-4573-b71f-25c0fc2f0660-iptables-alerter-script\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.547594    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547603    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-hosts-file\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547634    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-registration-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.547658    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:33:58.047638233 +0000 UTC m=+2.089686100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547681    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-registration-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547685    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-os-release\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547710    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-multus-certs\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547724    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-hostroot\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547748    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-host-run-multus-certs\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547800    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99cc38f8-6448-4047-a377-018ed5c08a5f-os-release\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547847    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-hosts-file\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.547972    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15520a7b-37bd-4094-9b6c-b542c254513d-socket-dir\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.548256    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d55cc1510d7fd2078798bc611e3e5c.slice/crio-08e49e4cfcde79533251a496f79939a3be72498b2b2fe41ac2772a345f334d85 WatchSource:0}: Error finding container 08e49e4cfcde79533251a496f79939a3be72498b2b2fe41ac2772a345f334d85: Status 404 returned error can't find the container with id 08e49e4cfcde79533251a496f79939a3be72498b2b2fe41ac2772a345f334d85
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.548301    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d72c928-1cc5-4573-b71f-25c0fc2f0660-iptables-alerter-script\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.549603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.548754    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99cc38f8-6448-4047-a377-018ed5c08a5f-multus-daemon-config\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.550188 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.548817    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9f9e23f5a0fe49a38ad742dd8ed6c0.slice/crio-f1660902b8dfc86c3f21464317debfca8c80f9e8f7c091edf29ff2361d9a0d3f WatchSource:0}: Error finding container f1660902b8dfc86c3f21464317debfca8c80f9e8f7c091edf29ff2361d9a0d3f: Status 404 returned error can't find the container with id f1660902b8dfc86c3f21464317debfca8c80f9e8f7c091edf29ff2361d9a0d3f
Jul 24 13:33:57.550188 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.549659    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-tmp\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.550188 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.550073    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-etc-tuned\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.556190 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.556176    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:33:57.559833 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.559775    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hkb\" (UniqueName: \"kubernetes.io/projected/15520a7b-37bd-4094-9b6c-b542c254513d-kube-api-access-k6hkb\") pod \"aws-ebs-csi-driver-node-m9p5j\" (UID: \"15520a7b-37bd-4094-9b6c-b542c254513d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.559833 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.559827    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knbd\" (UniqueName: \"kubernetes.io/projected/6d72c928-1cc5-4573-b71f-25c0fc2f0660-kube-api-access-7knbd\") pod \"iptables-alerter-hbxwj\" (UID: \"6d72c928-1cc5-4573-b71f-25c0fc2f0660\") " pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.560066 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.560048    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9qn\" (UniqueName: \"kubernetes.io/projected/9ac42bbb-5758-40f3-b2b2-99dbb52e44b2-kube-api-access-lc9qn\") pod \"tuned-ghlrl\" (UID: \"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.560126 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.560105    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbd8k\" (UniqueName: \"kubernetes.io/projected/99cc38f8-6448-4047-a377-018ed5c08a5f-kube-api-access-mbd8k\") pod \"multus-xqs49\" (UID: \"99cc38f8-6448-4047-a377-018ed5c08a5f\") " pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.561239 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.561213    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4f5\" (UniqueName: \"kubernetes.io/projected/a1e62ee6-1992-460f-ab28-59e9ceb5d89c-kube-api-access-dc4f5\") pod \"node-resolver-z26h4\" (UID: \"a1e62ee6-1992-460f-ab28-59e9ceb5d89c\") " pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.665632 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.665602    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:33:57.674375 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.674350    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961928b2_f95d_4775_88c0_c80ab72bd7be.slice/crio-c76e18dabd00814ac15e1156df7cd0ad86089f54f5fd4e3e74e4541a9fe5b28f WatchSource:0}: Error finding container c76e18dabd00814ac15e1156df7cd0ad86089f54f5fd4e3e74e4541a9fe5b28f: Status 404 returned error can't find the container with id c76e18dabd00814ac15e1156df7cd0ad86089f54f5fd4e3e74e4541a9fe5b28f
Jul 24 13:33:57.682609 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.682588    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:33:57.687523 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.687498    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9nhlx"
Jul 24 13:33:57.687964 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.687946    2527 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:33:57.689755 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.689721    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5be829c_ee49_4416_9f38_a6d3cd79c176.slice/crio-551e119d46439feaffcf1503986226d103fb32f71a49a4a992d3a65168688fcf WatchSource:0}: Error finding container 551e119d46439feaffcf1503986226d103fb32f71a49a4a992d3a65168688fcf: Status 404 returned error can't find the container with id 551e119d46439feaffcf1503986226d103fb32f71a49a4a992d3a65168688fcf
Jul 24 13:33:57.695331 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.695307    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8705eba6_6f23_4ad9_8a91_5cdee758f61a.slice/crio-c9dc2bef872a8a8650828a785b4b384362e4ee1f2f4e1b8e60c1ed5aa6550342 WatchSource:0}: Error finding container c9dc2bef872a8a8650828a785b4b384362e4ee1f2f4e1b8e60c1ed5aa6550342: Status 404 returned error can't find the container with id c9dc2bef872a8a8650828a785b4b384362e4ee1f2f4e1b8e60c1ed5aa6550342
Jul 24 13:33:57.698626 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.698609    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-szkt4"
Jul 24 13:33:57.705362 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.705345    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j"
Jul 24 13:33:57.706869 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.706839    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78698a3e_c8e7_473e_87e2_1fb4894d2c59.slice/crio-45fddc55b28a87d9a95f4729861fe8f010b4efcdf93930fa79fdb2e211f511b1 WatchSource:0}: Error finding container 45fddc55b28a87d9a95f4729861fe8f010b4efcdf93930fa79fdb2e211f511b1: Status 404 returned error can't find the container with id 45fddc55b28a87d9a95f4729861fe8f010b4efcdf93930fa79fdb2e211f511b1
Jul 24 13:33:57.712887 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.712858    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15520a7b_37bd_4094_9b6c_b542c254513d.slice/crio-d9e7575f9cfa6099f440877ffd6218ea73bac4b689ec4e210daa5acaa4071bf7 WatchSource:0}: Error finding container d9e7575f9cfa6099f440877ffd6218ea73bac4b689ec4e210daa5acaa4071bf7: Status 404 returned error can't find the container with id d9e7575f9cfa6099f440877ffd6218ea73bac4b689ec4e210daa5acaa4071bf7
Jul 24 13:33:57.732954 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.732922    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xqs49"
Jul 24 13:33:57.742366 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.742340    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cc38f8_6448_4047_a377_018ed5c08a5f.slice/crio-732377b071296e08d316ac70c148ad46a9a8cae7e73e83a2460321894884a095 WatchSource:0}: Error finding container 732377b071296e08d316ac70c148ad46a9a8cae7e73e83a2460321894884a095: Status 404 returned error can't find the container with id 732377b071296e08d316ac70c148ad46a9a8cae7e73e83a2460321894884a095
Jul 24 13:33:57.749688 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.749668    2527 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:33:57.755187 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.755170    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hbxwj"
Jul 24 13:33:57.762493 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.762471    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d72c928_1cc5_4573_b71f_25c0fc2f0660.slice/crio-81b1179c4f37787086e7393b8c25392bc2faf508c845de9690f46c3075f4860a WatchSource:0}: Error finding container 81b1179c4f37787086e7393b8c25392bc2faf508c845de9690f46c3075f4860a: Status 404 returned error can't find the container with id 81b1179c4f37787086e7393b8c25392bc2faf508c845de9690f46c3075f4860a
Jul 24 13:33:57.788220 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.788189    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlrl"
Jul 24 13:33:57.793533 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.793515    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z26h4"
Jul 24 13:33:57.795966 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.795944    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac42bbb_5758_40f3_b2b2_99dbb52e44b2.slice/crio-1a8e380f216cedc5487a1f5e642739aea158f4514b7c6ee45bd2e45076dd5faf WatchSource:0}: Error finding container 1a8e380f216cedc5487a1f5e642739aea158f4514b7c6ee45bd2e45076dd5faf: Status 404 returned error can't find the container with id 1a8e380f216cedc5487a1f5e642739aea158f4514b7c6ee45bd2e45076dd5faf
Jul 24 13:33:57.803617 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:33:57.803595    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e62ee6_1992_460f_ab28_59e9ceb5d89c.slice/crio-01f4df12601582ad66d923d2b24c13187ee62cb291aa702a31d06580c7aa58a6 WatchSource:0}: Error finding container 01f4df12601582ad66d923d2b24c13187ee62cb291aa702a31d06580c7aa58a6: Status 404 returned error can't find the container with id 01f4df12601582ad66d923d2b24c13187ee62cb291aa702a31d06580c7aa58a6
Jul 24 13:33:57.950867 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:57.950831    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:57.951052 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.950951    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:57.951052 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:57.951009    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:33:58.950994336 +0000 UTC m=+2.993042201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:58.052122 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.052075    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:58.052286 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.052189    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:58.052411 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.052394    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:33:58.052505 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.052425    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:33:58.052505 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.052439    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:58.052632 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.052520    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:33:59.052479998 +0000 UTC m=+3.094527868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:58.052978 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.052951    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:58.053115 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.053009    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:33:59.052993316 +0000 UTC m=+3.095041199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:58.417877 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.417807    2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:28:57 +0000 UTC" deadline="2028-03-25 15:05:08.391340316 +0000 UTC"
Jul 24 13:33:58.417877 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.417859    2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14641h31m9.973484475s"
Jul 24 13:33:58.536219 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.536096    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z26h4" event={"ID":"a1e62ee6-1992-460f-ab28-59e9ceb5d89c","Type":"ContainerStarted","Data":"01f4df12601582ad66d923d2b24c13187ee62cb291aa702a31d06580c7aa58a6"}
Jul 24 13:33:58.555313 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.555214    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nhlx" event={"ID":"8705eba6-6f23-4ad9-8a91-5cdee758f61a","Type":"ContainerStarted","Data":"c9dc2bef872a8a8650828a785b4b384362e4ee1f2f4e1b8e60c1ed5aa6550342"}
Jul 24 13:33:58.562038 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.561892    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"551e119d46439feaffcf1503986226d103fb32f71a49a4a992d3a65168688fcf"}
Jul 24 13:33:58.574871 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.574767    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"44d55cc1510d7fd2078798bc611e3e5c","Type":"ContainerStarted","Data":"08e49e4cfcde79533251a496f79939a3be72498b2b2fe41ac2772a345f334d85"}
Jul 24 13:33:58.606787 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.606656    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlrl" event={"ID":"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2","Type":"ContainerStarted","Data":"1a8e380f216cedc5487a1f5e642739aea158f4514b7c6ee45bd2e45076dd5faf"}
Jul 24 13:33:58.637935 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.637821    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hbxwj" event={"ID":"6d72c928-1cc5-4573-b71f-25c0fc2f0660","Type":"ContainerStarted","Data":"81b1179c4f37787086e7393b8c25392bc2faf508c845de9690f46c3075f4860a"}
Jul 24 13:33:58.645804 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.645753    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xqs49" event={"ID":"99cc38f8-6448-4047-a377-018ed5c08a5f","Type":"ContainerStarted","Data":"732377b071296e08d316ac70c148ad46a9a8cae7e73e83a2460321894884a095"}
Jul 24 13:33:58.658681 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.658626    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j" event={"ID":"15520a7b-37bd-4094-9b6c-b542c254513d","Type":"ContainerStarted","Data":"d9e7575f9cfa6099f440877ffd6218ea73bac4b689ec4e210daa5acaa4071bf7"}
Jul 24 13:33:58.664980 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.664937    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerStarted","Data":"45fddc55b28a87d9a95f4729861fe8f010b4efcdf93930fa79fdb2e211f511b1"}
Jul 24 13:33:58.674543 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.674418    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnqt6" event={"ID":"961928b2-f95d-4775-88c0-c80ab72bd7be","Type":"ContainerStarted","Data":"c76e18dabd00814ac15e1156df7cd0ad86089f54f5fd4e3e74e4541a9fe5b28f"}
Jul 24 13:33:58.692780 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.692735    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"2f9f9e23f5a0fe49a38ad742dd8ed6c0","Type":"ContainerStarted","Data":"f1660902b8dfc86c3f21464317debfca8c80f9e8f7c091edf29ff2361d9a0d3f"}
Jul 24 13:33:58.854023 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.853425    2527 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:33:58.964490 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:58.963721    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:58.964490 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.963982    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:58.964490 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:58.964055    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:00.964034488 +0000 UTC m=+5.006082369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:33:59.064988 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.064948    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:59.065171 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.065044    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:59.065529 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065237    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:59.065529 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065367    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:34:01.065294712 +0000 UTC m=+5.107342594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:33:59.065529 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065377    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:33:59.065529 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065406    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:33:59.065529 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065418    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:59.065969 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.065934    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:01.065455666 +0000 UTC m=+5.107503540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:33:59.419124 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.419071    2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:28:57 +0000 UTC" deadline="2028-02-07 15:51:24.303770184 +0000 UTC"
Jul 24 13:33:59.419124 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.419123    2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13514h17m24.884652901s"
Jul 24 13:33:59.507636 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.507596    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:33:59.507838 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.507752    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:33:59.508375 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.508349    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:33:59.508532 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.508474    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:33:59.508615 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:33:59.508584    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:33:59.508721 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:33:59.508681    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:00.238217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:00.237959    2527 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:00.986875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:00.986835    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:00.987426 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:00.987028    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:00.987426 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:00.987095    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:04.987076471 +0000 UTC m=+9.029124342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:01.088215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:01.088172    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:01.088382 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:01.088240    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:01.088450 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.088405    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:01.088533 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.088491    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:34:05.088453316 +0000 UTC m=+9.130501195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:01.088979 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.088954    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:01.088979 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.088982    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:01.089134 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.088995    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:01.089134 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.089056    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:05.089031094 +0000 UTC m=+9.131078962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:01.507941 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:01.507888    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:01.508151 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.508069    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:01.508627 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:01.508592    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:01.508741 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.508707    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:01.508807 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:01.508758    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:01.508915 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:01.508877    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:03.508228    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:03.508345    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:03.508751    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:03.508854    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:03.508910    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:03.509019 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:03.508981    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:05.024586 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.023949    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:05.024586 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.024150    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:05.024586 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.024213    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:13.02419565 +0000 UTC m=+17.066243528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:05.124944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.124901    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:05.125131 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.124956    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:05.125131 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125099    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:05.125131 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125101    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:05.125131 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125126    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:05.125339 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125137    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:05.125339 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125159    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:34:13.125141686 +0000 UTC m=+17.167189570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:05.125339 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.125186    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:13.125173467 +0000 UTC m=+17.167221333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:05.507777 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.507676    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:05.507777 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.507713    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:05.507777 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:05.507721    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:05.508034 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.507806    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:05.508034 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.507903    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:05.508034 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:05.507997    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:07.507782 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:07.507710    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:07.507782 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:07.507753    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:07.508311 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:07.507844    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:07.508311 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:07.508119    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:07.508311 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:07.508190    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:07.508311 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:07.508262    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:09.507950 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:09.507910    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:09.508379 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:09.507920    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:09.508379 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:09.508030    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:09.508379 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:09.507919    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:09.508379 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:09.508134    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:09.508379 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:09.508226    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:11.508367 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:11.508325    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:11.508367 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:11.508371    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:11.508831 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:11.508347    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:11.508831 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:11.508483    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:11.508831 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:11.508572    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:11.508831 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:11.508641    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:13.086554 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.086310    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:13.086960 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.086498    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:13.086960 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.086626    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:29.086608851 +0000 UTC m=+33.128656713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:13.187801 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.187767    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:13.187971 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.187818    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:13.187971 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.187932    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:13.187971 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.187949    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:13.187971 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.187965    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:13.187971 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.187976    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:13.188180 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.187993    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:34:29.187975406 +0000 UTC m=+33.230023272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:13.188180 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.188009    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:29.187999992 +0000 UTC m=+33.230047854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:13.508000 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.507919    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:13.508000 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.507965    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:13.508223 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.508028    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:13.508223 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:13.508134    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:13.508312 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.508238    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:13.508362 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:13.508320    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:15.507884 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:15.507840    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:15.508362 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:15.507856    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:15.508362 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:15.507958    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:15.508362 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:15.507859    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:15.508362 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:15.508031    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:15.508362 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:15.508089    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:17.507808 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:17.507768    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:17.508210 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:17.507767    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:17.508210 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:17.507907    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:17.508210 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:17.507767    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:17.508210 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:17.507984    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:17.508210 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:17.508048    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:18.742097 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.742053    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlrl" event={"ID":"9ac42bbb-5758-40f3-b2b2-99dbb52e44b2","Type":"ContainerStarted","Data":"975c75374d6b97c0ac889907d01e55a9983079ad4782c5bbc748328fe37b43c8"}
Jul 24 13:34:18.743573 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.743548    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xqs49" event={"ID":"99cc38f8-6448-4047-a377-018ed5c08a5f","Type":"ContainerStarted","Data":"4a6bd0d9e93bb8921ce9b5e176bba610eccc1518852733087a4b64d0a8a9cad3"}
Jul 24 13:34:18.744852 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.744827    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j" event={"ID":"15520a7b-37bd-4094-9b6c-b542c254513d","Type":"ContainerStarted","Data":"9cf1b26d8c8617c82da443fc4f6162e6598626dfb0a5beb3551876a6316f9240"}
Jul 24 13:34:18.745901 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.745871    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerStarted","Data":"9c9714b82f2ba556b2f32b59ca29a366631a32b55b0adf281e45fad387a89d4d"}
Jul 24 13:34:18.747141 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.747110    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qnqt6" event={"ID":"961928b2-f95d-4775-88c0-c80ab72bd7be","Type":"ContainerStarted","Data":"2cab53c08a2d5c229345afcb2f8c40f44dcb1728fd438c0b7fe145347af37bfc"}
Jul 24 13:34:18.748373 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.748352    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"2f9f9e23f5a0fe49a38ad742dd8ed6c0","Type":"ContainerStarted","Data":"71e16292b96f1ef2184b1815dc95747a3098a18ced8a2cad748121bdd5dbb624"}
Jul 24 13:34:18.748531 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.748517    2527 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"
Jul 24 13:34:18.749694 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.749667    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z26h4" event={"ID":"a1e62ee6-1992-460f-ab28-59e9ceb5d89c","Type":"ContainerStarted","Data":"90d853d97e859766dd6a1b7c6a403133603a75787761eff12d306981f80adabb"}
Jul 24 13:34:18.751188 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.751165    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9nhlx" event={"ID":"8705eba6-6f23-4ad9-8a91-5cdee758f61a","Type":"ContainerStarted","Data":"850b91f144ac411603f5049fbaeb258e2fe2ccd7fac447867cb20a71665e526e"}
Jul 24 13:34:18.753478 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.753457    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"ad851d347bbb9efa5cadf5c4fb59ed569cf3ee08bcc2aa6d84d301b9c1f637f7"}
Jul 24 13:34:18.753568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.753525    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"1adac960d2e71c3094c5365cec61458f8965cc86c2569fd713a7cc71505e5662"}
Jul 24 13:34:18.753621 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.753568    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"0dcc603dcb997ab3f871cca02e6e6e13a729f709653f6b0badbd557dd529e4dd"}
Jul 24 13:34:18.754893 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.754872    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"44d55cc1510d7fd2078798bc611e3e5c","Type":"ContainerStarted","Data":"08d1e847195dd6f836fba802c5c125ec55ecda6a592a6ed70227e27abad27c85"}
Jul 24 13:34:18.775319 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.775294    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"]
Jul 24 13:34:18.775428 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.775406    2527 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:34:18.819695 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.819424    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ghlrl" podStartSLOduration=1.329137431 podStartE2EDuration="21.819410039s" totalImagesPullingTime="20.490272608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:18.819032205 +0000 UTC m=+22.861080092" watchObservedRunningTime="2026-07-24 13:34:18.819410039 +0000 UTC m=+22.861457923"
Jul 24 13:34:18.928631 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.928587    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z26h4" podStartSLOduration=1.427745741 podStartE2EDuration="21.9285699s" totalImagesPullingTime="20.500824159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:18.926280975 +0000 UTC m=+22.968328861" watchObservedRunningTime="2026-07-24 13:34:18.9285699 +0000 UTC m=+22.970617785"
Jul 24 13:34:18.928924 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.928905    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xqs49" podStartSLOduration=1.319179608 podStartE2EDuration="21.928899268s" totalImagesPullingTime="20.60971966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:18.896621176 +0000 UTC m=+22.938669062" watchObservedRunningTime="2026-07-24 13:34:18.928899268 +0000 UTC m=+22.970947156"
Jul 24 13:34:18.990789 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:18.990741    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" podStartSLOduration=0.990726682 podStartE2EDuration="990.726682ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:18.990154849 +0000 UTC m=+23.032202734" watchObservedRunningTime="2026-07-24 13:34:18.990726682 +0000 UTC m=+23.032774566"
Jul 24 13:34:19.073195 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.073153    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9nhlx" podStartSLOduration=1.473769823 podStartE2EDuration="22.073136906s" totalImagesPullingTime="20.599367083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:19.033878904 +0000 UTC m=+23.075926791" watchObservedRunningTime="2026-07-24 13:34:19.073136906 +0000 UTC m=+23.115184791"
Jul 24 13:34:19.508049 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.508009    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:19.508215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.508016    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:19.508215 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:19.508119    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:19.508215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.508158    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:19.508375 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:19.508241    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:19.508375 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:19.508354    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:19.758108 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.758061    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="9c9714b82f2ba556b2f32b59ca29a366631a32b55b0adf281e45fad387a89d4d" exitCode=0
Jul 24 13:34:19.758654 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.758153    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"9c9714b82f2ba556b2f32b59ca29a366631a32b55b0adf281e45fad387a89d4d"}
Jul 24 13:34:19.761484 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.761461    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"141f0655b35b8141238660e9263b9edafe80a1ee491e4655dbd89f2a7e924c8b"}
Jul 24 13:34:19.761605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.761536    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"06bf4679e701b0efde3d83a1f2a3a5ad2494f3fc42ce3f6673015d6ef50a5eee"}
Jul 24 13:34:19.761605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.761582    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"a7887e1386eb39cb8f9cb19479d9b064fcf7bd5e1371cad52cd8db84707ab6da"}
Jul 24 13:34:19.762996 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.762976    2527 generic.go:356] "Generic (PLEG): container finished" podID="44d55cc1510d7fd2078798bc611e3e5c" containerID="08d1e847195dd6f836fba802c5c125ec55ecda6a592a6ed70227e27abad27c85" exitCode=0
Jul 24 13:34:19.763094 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.763069    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"44d55cc1510d7fd2078798bc611e3e5c","Type":"ContainerDied","Data":"08d1e847195dd6f836fba802c5c125ec55ecda6a592a6ed70227e27abad27c85"}
Jul 24 13:34:19.763155 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.763115    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"44d55cc1510d7fd2078798bc611e3e5c","Type":"ContainerStarted","Data":"2cfa7c04f322172715f1af4481f9ff6b51148ea169033797047c242cf7ef0bce"}
Jul 24 13:34:19.786842 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.786789    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qnqt6" podStartSLOduration=3.164051658 podStartE2EDuration="23.786776257s" totalImagesPullingTime="20.622724599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:19.108894622 +0000 UTC m=+23.150942499" watchObservedRunningTime="2026-07-24 13:34:19.786776257 +0000 UTC m=+23.828824142"
Jul 24 13:34:19.806363 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:19.806301    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" podStartSLOduration=22.806280044 podStartE2EDuration="22.806280044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:19.806036242 +0000 UTC m=+23.848084129" watchObservedRunningTime="2026-07-24 13:34:19.806280044 +0000 UTC m=+23.848327924"
Jul 24 13:34:20.455403 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.455376    2527 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 13:34:20.767555 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.767453    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hbxwj" event={"ID":"6d72c928-1cc5-4573-b71f-25c0fc2f0660","Type":"ContainerStarted","Data":"5619b5c4c661f7a6fccb43e41ebb3f7c370dd0322bf99a636b2de981c5da66fb"}
Jul 24 13:34:20.769632 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.769604    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j" event={"ID":"15520a7b-37bd-4094-9b6c-b542c254513d","Type":"ContainerStarted","Data":"d40f7927feec10876f18155ab763a21472ae9f5767ec564458d4072ee01d515e"}
Jul 24 13:34:20.780612 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.780587    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:34:20.781295 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.781273    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:34:20.782739 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:20.782694    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hbxwj" podStartSLOduration=3.246558208 podStartE2EDuration="23.782677048s" totalImagesPullingTime="20.53611884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:20.782375698 +0000 UTC m=+24.824423583" watchObservedRunningTime="2026-07-24 13:34:20.782677048 +0000 UTC m=+24.824724934"
Jul 24 13:34:21.420104 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.419950    2527 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T13:34:20.45539724Z","UUID":"7f87cf0c-a398-4b99-9a25-fcb2490dcfcc","Handler":null,"Name":"","Endpoint":""}
Jul 24 13:34:21.421971 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.421943    2527 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 13:34:21.422084 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.421998    2527 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 13:34:21.508271 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.508227    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:21.508466 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.508346    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:21.508466 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:21.508356    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:21.508466 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.508374    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:21.508466 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:21.508449    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:21.508692 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:21.508530    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:21.775731 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.775643    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"cae75c50c8f71cb38b1992caa3baeb425e627e119761f4be2732cc8053df481e"}
Jul 24 13:34:21.776231 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.775828    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:34:21.776377 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:21.776362    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qnqt6"
Jul 24 13:34:22.780130 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:22.780097    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j" event={"ID":"15520a7b-37bd-4094-9b6c-b542c254513d","Type":"ContainerStarted","Data":"c77813492dc1a44ed159db8aa2413ea43a18f3a6d323adf2e9f58566b6858f52"}
Jul 24 13:34:22.803813 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:22.803763    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9p5j" podStartSLOduration=2.222196973 podStartE2EDuration="25.803747997s" totalImagesPullingTime="23.581551024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:22.803016264 +0000 UTC m=+26.845064149" watchObservedRunningTime="2026-07-24 13:34:22.803747997 +0000 UTC m=+26.845795933"
Jul 24 13:34:23.507519 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:23.507478    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:23.507700 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:23.507478    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:23.507700 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:23.507610    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:23.507792 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:23.507711    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:23.507792 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:23.507479    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:23.507792 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:23.507787    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:25.508114 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.507873    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:25.508933 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.507880    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:25.508933 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:25.508164    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:25.508933 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.507921    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:25.508933 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:25.508241    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:25.508933 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:25.508307    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:25.788810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.788732    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="99ac62f8e59045672ef7f2bf35268fc9e87edaaa2465d8fb8628cd5970b6de0d" exitCode=0
Jul 24 13:34:25.788985 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.788811    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"99ac62f8e59045672ef7f2bf35268fc9e87edaaa2465d8fb8628cd5970b6de0d"}
Jul 24 13:34:25.792770 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.792748    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" event={"ID":"b5be829c-ee49-4416-9f38-a6d3cd79c176","Type":"ContainerStarted","Data":"68e6ebac4ac488310e0385f16996534a2c996ad458f29057ee85635b59601179"}
Jul 24 13:34:25.793031 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.793009    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:25.793197 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.793080    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:25.793197 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.793127    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:25.809508 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.809481    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:25.809673 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.809614    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:25.842113 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:25.842062    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58" podStartSLOduration=9.177585433 podStartE2EDuration="29.842046791s" totalImagesPullingTime="20.664461358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:25.841734282 +0000 UTC m=+29.883782194" watchObservedRunningTime="2026-07-24 13:34:25.842046791 +0000 UTC m=+29.884094676"
Jul 24 13:34:27.234236 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.233977    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qq8rj"]
Jul 24 13:34:27.234623 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.234282    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:27.234623 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:27.234368    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:27.238474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.238449    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdskt"]
Jul 24 13:34:27.238624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.238558    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:27.238698 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:27.238677    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:27.252526 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.252493    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rj5bn"]
Jul 24 13:34:27.252654 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.252598    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:27.252714 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:27.252676    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:27.799885 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.799853    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="05204117e0a6d9062015b33a0530c850ae31552242a11b5055bc3b6d1637ab83" exitCode=0
Jul 24 13:34:27.800067 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:27.799931    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"05204117e0a6d9062015b33a0530c850ae31552242a11b5055bc3b6d1637ab83"}
Jul 24 13:34:28.508409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:28.508360    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:28.508409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:28.508395    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:28.508985 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:28.508494    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:28.508985 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:28.508632    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:28.804565 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:28.804424    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="bbf56b9b0d7f84d984ed288919c7623525a314f9523ab2861fe61ee2538488e0" exitCode=0
Jul 24 13:34:28.804565 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:28.804483    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"bbf56b9b0d7f84d984ed288919c7623525a314f9523ab2861fe61ee2538488e0"}
Jul 24 13:34:29.108561 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:29.108518    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:29.108719 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.108696    2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:29.108775 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.108769    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs podName:8b318dd6-b05b-4f18-abc7-6af9858c0690 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:01.108753567 +0000 UTC m=+65.150801433 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs") pod "network-metrics-daemon-gdskt" (UID: "8b318dd6-b05b-4f18-abc7-6af9858c0690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:29.209263 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:29.209222    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:29.209263 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:29.209262    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209364    2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209383    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209403    2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209414    2527 projected.go:196] Error preparing data for projected volume kube-api-access-gqzlr for pod openshift-network-diagnostics/network-check-target-rj5bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209421    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret podName:ac3c72de-350d-4005-9d29-bf40ae47ac5c nodeName:}" failed. No retries permitted until 2026-07-24 13:35:01.209405275 +0000 UTC m=+65.251453142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret") pod "global-pull-secret-syncer-qq8rj" (UID: "ac3c72de-350d-4005-9d29-bf40ae47ac5c") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:29.209481 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.209483    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr podName:33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:01.209467185 +0000 UTC m=+65.251515047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqzlr" (UniqueName: "kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr") pod "network-check-target-rj5bn" (UID: "33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:29.507509 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:29.507460    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:29.507688 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:29.507616    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gdskt" podUID="8b318dd6-b05b-4f18-abc7-6af9858c0690"
Jul 24 13:34:30.507878 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:30.507839    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:30.508268 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:30.507892    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:30.508268 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:30.507984    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qq8rj" podUID="ac3c72de-350d-4005-9d29-bf40ae47ac5c"
Jul 24 13:34:30.508268 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:30.508082    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rj5bn" podUID="33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6"
Jul 24 13:34:31.254103 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.254032    2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeReady"
Jul 24 13:34:31.254245 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.254176    2527 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 13:34:31.301700 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.301662    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:34:31.303817 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.303797    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.305865 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.305803    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 13:34:31.305865 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.305817    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 13:34:31.306254 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.306234    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"]
Jul 24 13:34:31.306697 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.306640    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 13:34:31.306993 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.306963    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hklft\""
Jul 24 13:34:31.308055 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.308035    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.310927 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.310905    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.312033 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.312012    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x2wfq\""
Jul 24 13:34:31.312257 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.312240    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.312742 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.312725    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 13:34:31.314528 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.314508    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 13:34:31.315294 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.315252    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 13:34:31.317950 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.317928    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"]
Jul 24 13:34:31.319872 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.319853    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.320574 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.320106    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d4b8fd45f-z8bhm"]
Jul 24 13:34:31.326084 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.326047    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"]
Jul 24 13:34:31.326474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.326453    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.327702 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.327684    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"]
Jul 24 13:34:31.327842 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.327822    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.329527 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.329508    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xqgfh"]
Jul 24 13:34:31.329645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.329628    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.331343 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.331278    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7896g"]
Jul 24 13:34:31.331473 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.331425    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.333669 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.333635    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.344388 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.344365    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:34:31.344572 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.344419    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"]
Jul 24 13:34:31.347231 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347209    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.347642 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347615    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.347642 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347615    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 13:34:31.347882 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347709    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 13:34:31.347882 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347726    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.347882 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347716    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 13:34:31.348018 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.347942    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.348184 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348164    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 13:34:31.348271 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348217    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.348271 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348238    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 24 13:34:31.348361 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348284    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 13:34:31.348420 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348406    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jd4l2\""
Jul 24 13:34:31.348631 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348613    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.348710 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348669    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 13:34:31.348710 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348677    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p5ntg\""
Jul 24 13:34:31.348782 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348749    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q5m7n\""
Jul 24 13:34:31.348930 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348903    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 24 13:34:31.349093 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.349074    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.349288 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.349193    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 13:34:31.349288 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.349268    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 13:34:31.349387 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.349284    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.349588 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348957    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9bqww\""
Jul 24 13:34:31.349838 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.348991    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dvmkc\""
Jul 24 13:34:31.350680 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.350123    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.350680 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.350159    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dfjpt\""
Jul 24 13:34:31.350680 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.350284    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 13:34:31.361222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.361193    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 13:34:31.361768 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.361749    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.361986 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.361965    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.362711 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.362691    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.362801 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.362717    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 13:34:31.367383 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.367362    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"]
Jul 24 13:34:31.376621 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.376600    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 13:34:31.376816 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.376624    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"]
Jul 24 13:34:31.377884 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.377863    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d4b8fd45f-z8bhm"]
Jul 24 13:34:31.378540 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.378513    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"]
Jul 24 13:34:31.382351 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.382325    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 24 13:34:31.382498 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.382457    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xqgfh"]
Jul 24 13:34:31.382682 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.382666    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"
Jul 24 13:34:31.384330 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.384306    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"]
Jul 24 13:34:31.387193 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.387165    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7896g"]
Jul 24 13:34:31.387310 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.387296    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"
Jul 24 13:34:31.390147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.388736    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8xhb2\""
Jul 24 13:34:31.390147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.389486    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.390147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.389872    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lw57p\""
Jul 24 13:34:31.390336 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.390265    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"]
Jul 24 13:34:31.390370 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.390348    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.390571 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.390545    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.392329 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.392306    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.409176 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.409149    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"]
Jul 24 13:34:31.424349 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424316    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplbk\" (UniqueName: \"kubernetes.io/projected/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-kube-api-access-nplbk\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.424551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424362    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b2a17c-563e-4e60-868b-41f51530c308-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.424551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424417    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1587d831-aefd-4241-8f63-e0a31867f65d-serving-cert\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.424551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424469    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/1587d831-aefd-4241-8f63-e0a31867f65d-kube-api-access-dq7c2\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.424551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424497    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424562    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424603    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5446l\" (UniqueName: \"kubernetes.io/projected/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-kube-api-access-5446l\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424629    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-default-certificate\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424693    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-config\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424722    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80506080-1458-4891-ba70-00afb1e62048-config\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.424753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424741    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487fd\" (UniqueName: \"kubernetes.io/projected/c02748a9-e0c5-4d09-93e2-5d65dd400703-kube-api-access-487fd\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424760    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424798    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rmp\" (UniqueName: \"kubernetes.io/projected/80506080-1458-4891-ba70-00afb1e62048-kube-api-access-q7rmp\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424858    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424903    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424923    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.424960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424939    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.424962    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-tmp\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425004    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-snapshots\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425032    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d308633-b904-498a-b0b1-f8ca0a66fd0f-serving-cert\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425060    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425136    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-service-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425169    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-stats-auth\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.425222 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425196    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwpm\" (UniqueName: \"kubernetes.io/projected/65b2a17c-563e-4e60-868b-41f51530c308-kube-api-access-szwpm\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425243    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-trusted-ca\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425283    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425313    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4982t\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425348    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65b2a17c-563e-4e60-868b-41f51530c308-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425389    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80506080-1458-4891-ba70-00afb1e62048-serving-cert\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425408    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425462    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425490    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425508    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.425583 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.425523    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcst6\" (UniqueName: \"kubernetes.io/projected/6d308633-b904-498a-b0b1-f8ca0a66fd0f-kube-api-access-tcst6\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.438006 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.437976    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hggbw"]
Jul 24 13:34:31.440152 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.440131    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c946z"]
Jul 24 13:34:31.440333 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.440292    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.442002 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.441973    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"]
Jul 24 13:34:31.442109 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.442055    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.446339 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.446318    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 13:34:31.446687 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.446653    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 13:34:31.447403 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.447385    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 13:34:31.447531 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.447458    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h9qpm\""
Jul 24 13:34:31.447652 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.447609    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 13:34:31.447725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.447708    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kzzx2\""
Jul 24 13:34:31.447859 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.447837    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 13:34:31.456468 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.456356    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hggbw"]
Jul 24 13:34:31.475343 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.475311    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c946z"]
Jul 24 13:34:31.507775 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.507692    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:34:31.513335 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.513307    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:34:31.513335 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.513313    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8g8q\""
Jul 24 13:34:31.526825 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526794    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49397988-6b3a-4429-a7c0-b700fd2b1088-cert\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526830    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80506080-1458-4891-ba70-00afb1e62048-config\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526862    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487fd\" (UniqueName: \"kubernetes.io/projected/c02748a9-e0c5-4d09-93e2-5d65dd400703-kube-api-access-487fd\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526888    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526914    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rmp\" (UniqueName: \"kubernetes.io/projected/80506080-1458-4891-ba70-00afb1e62048-kube-api-access-q7rmp\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526944    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526971    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.526995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.526992    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527014    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527038    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4p2\" (UniqueName: \"kubernetes.io/projected/e7d55935-92b8-48bb-93de-593c41e1c7d3-kube-api-access-qx4p2\") pod \"network-check-source-7d88cb55d8-5x9s7\" (UID: \"e7d55935-92b8-48bb-93de-593c41e1c7d3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527062    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-tmp\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527084    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-config-volume\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527110    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-snapshots\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527133    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d308633-b904-498a-b0b1-f8ca0a66fd0f-serving-cert\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527159    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527202    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-service-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.527255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527235    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-stats-auth\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527257    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwpm\" (UniqueName: \"kubernetes.io/projected/65b2a17c-563e-4e60-868b-41f51530c308-kube-api-access-szwpm\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527284    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-tmp-dir\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527304    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq5z\" (UniqueName: \"kubernetes.io/projected/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-kube-api-access-4nq5z\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527331    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-trusted-ca\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527357    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmkr\" (UniqueName: \"kubernetes.io/projected/4b8d0646-bb00-4c87-a2c8-4f5ece6de933-kube-api-access-vdmkr\") pod \"volume-data-source-validator-549c6d54cf-r9pgq\" (UID: \"4b8d0646-bb00-4c87-a2c8-4f5ece6de933\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527385    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527413    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4982t\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527469    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65b2a17c-563e-4e60-868b-41f51530c308-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527485    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80506080-1458-4891-ba70-00afb1e62048-config\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527527    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80506080-1458-4891-ba70-00afb1e62048-serving-cert\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527556    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527589    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.527629 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527614    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527637    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527660    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcst6\" (UniqueName: \"kubernetes.io/projected/6d308633-b904-498a-b0b1-f8ca0a66fd0f-kube-api-access-tcst6\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527693    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplbk\" (UniqueName: \"kubernetes.io/projected/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-kube-api-access-nplbk\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527718    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b2a17c-563e-4e60-868b-41f51530c308-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527749    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527788    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1587d831-aefd-4241-8f63-e0a31867f65d-serving-cert\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527817    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/1587d831-aefd-4241-8f63-e0a31867f65d-kube-api-access-dq7c2\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527843    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527872    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdw9k\" (UniqueName: \"kubernetes.io/projected/49397988-6b3a-4429-a7c0-b700fd2b1088-kube-api-access-vdw9k\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527905    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527935    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5446l\" (UniqueName: \"kubernetes.io/projected/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-kube-api-access-5446l\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.527961    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-default-certificate\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.528006    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-config\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.528212 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.528038    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.028023608 +0000 UTC m=+36.070071502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.528264    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.528514    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.528530    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.528562    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.028547784 +0000 UTC m=+36.070595687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.528730    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-config\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.528816    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:31.528875 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.528868    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.028853576 +0000 UTC m=+36.070901456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:31.529212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.529189    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.529355 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.529332    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65b2a17c-563e-4e60-868b-41f51530c308-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.529423 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.529377    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.529581 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.529547    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d308633-b904-498a-b0b1-f8ca0a66fd0f-trusted-ca\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.529709 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.529693    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:31.529766 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.529740    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.029726757 +0000 UTC m=+36.071774631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:31.529827 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.529809    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:31.529827 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.529819    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:31.529918 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.529851    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.029840005 +0000 UTC m=+36.071887886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:31.531102 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.530011    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-snapshots\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.531102 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.530198    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1587d831-aefd-4241-8f63-e0a31867f65d-tmp\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.531102 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.530518    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1587d831-aefd-4241-8f63-e0a31867f65d-service-ca-bundle\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.531102 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.530827    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.533152 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.533131    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65b2a17c-563e-4e60-868b-41f51530c308-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.534047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.533990    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d308633-b904-498a-b0b1-f8ca0a66fd0f-serving-cert\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.534047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.533999    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.534169 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.534049    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1587d831-aefd-4241-8f63-e0a31867f65d-serving-cert\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.534306 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.534283    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-stats-auth\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.535893 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.535848    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-default-certificate\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.536414 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.536390    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.536593 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.536524    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80506080-1458-4891-ba70-00afb1e62048-serving-cert\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.546903 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.546873    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwpm\" (UniqueName: \"kubernetes.io/projected/65b2a17c-563e-4e60-868b-41f51530c308-kube-api-access-szwpm\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf\" (UID: \"65b2a17c-563e-4e60-868b-41f51530c308\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.554065 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.553488    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7c2\" (UniqueName: \"kubernetes.io/projected/1587d831-aefd-4241-8f63-e0a31867f65d-kube-api-access-dq7c2\") pod \"insights-operator-5885fdb759-7896g\" (UID: \"1587d831-aefd-4241-8f63-e0a31867f65d\") " pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.554065 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.553547    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4982t\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.554065 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.554020    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rmp\" (UniqueName: \"kubernetes.io/projected/80506080-1458-4891-ba70-00afb1e62048-kube-api-access-q7rmp\") pod \"service-ca-operator-5d6c75f84f-s7xjd\" (UID: \"80506080-1458-4891-ba70-00afb1e62048\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.555490 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.555469    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:31.556355 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.556306    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-487fd\" (UniqueName: \"kubernetes.io/projected/c02748a9-e0c5-4d09-93e2-5d65dd400703-kube-api-access-487fd\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:31.556484 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.556387    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5446l\" (UniqueName: \"kubernetes.io/projected/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-kube-api-access-5446l\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:31.556632 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.556609    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcst6\" (UniqueName: \"kubernetes.io/projected/6d308633-b904-498a-b0b1-f8ca0a66fd0f-kube-api-access-tcst6\") pod \"console-operator-584dcb6b54-xqgfh\" (UID: \"6d308633-b904-498a-b0b1-f8ca0a66fd0f\") " pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.558480 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.558415    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplbk\" (UniqueName: \"kubernetes.io/projected/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-kube-api-access-nplbk\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:31.628758 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628723    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49397988-6b3a-4429-a7c0-b700fd2b1088-cert\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628782    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4p2\" (UniqueName: \"kubernetes.io/projected/e7d55935-92b8-48bb-93de-593c41e1c7d3-kube-api-access-qx4p2\") pod \"network-check-source-7d88cb55d8-5x9s7\" (UID: \"e7d55935-92b8-48bb-93de-593c41e1c7d3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628809    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-config-volume\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628853    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-tmp-dir\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628871    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq5z\" (UniqueName: \"kubernetes.io/projected/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-kube-api-access-4nq5z\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628888    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmkr\" (UniqueName: \"kubernetes.io/projected/4b8d0646-bb00-4c87-a2c8-4f5ece6de933-kube-api-access-vdmkr\") pod \"volume-data-source-validator-549c6d54cf-r9pgq\" (UID: \"4b8d0646-bb00-4c87-a2c8-4f5ece6de933\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"
Jul 24 13:34:31.628959 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628955    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.629274 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.628985    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdw9k\" (UniqueName: \"kubernetes.io/projected/49397988-6b3a-4429-a7c0-b700fd2b1088-kube-api-access-vdw9k\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.629413 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.629386    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-tmp-dir\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.629660 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.629641    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:31.629745 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:31.629711    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:32.129690094 +0000 UTC m=+36.171737978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:31.630226 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.630205    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-config-volume\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.632447 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.632408    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49397988-6b3a-4429-a7c0-b700fd2b1088-cert\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.636044 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.636024    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"
Jul 24 13:34:31.647334 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.647308    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdw9k\" (UniqueName: \"kubernetes.io/projected/49397988-6b3a-4429-a7c0-b700fd2b1088-kube-api-access-vdw9k\") pod \"ingress-canary-hggbw\" (UID: \"49397988-6b3a-4429-a7c0-b700fd2b1088\") " pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.647551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.647532    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmkr\" (UniqueName: \"kubernetes.io/projected/4b8d0646-bb00-4c87-a2c8-4f5ece6de933-kube-api-access-vdmkr\") pod \"volume-data-source-validator-549c6d54cf-r9pgq\" (UID: \"4b8d0646-bb00-4c87-a2c8-4f5ece6de933\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"
Jul 24 13:34:31.652404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.652377    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4p2\" (UniqueName: \"kubernetes.io/projected/e7d55935-92b8-48bb-93de-593c41e1c7d3-kube-api-access-qx4p2\") pod \"network-check-source-7d88cb55d8-5x9s7\" (UID: \"e7d55935-92b8-48bb-93de-593c41e1c7d3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"
Jul 24 13:34:31.653516 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.653493    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"
Jul 24 13:34:31.653753 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.653735    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq5z\" (UniqueName: \"kubernetes.io/projected/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-kube-api-access-4nq5z\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:31.676085 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.676003    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:31.677881 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.677844    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7896g"
Jul 24 13:34:31.698978 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.698743    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"
Jul 24 13:34:31.704596 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.704562    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"
Jul 24 13:34:31.733611 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.733554    2527 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qnqt6_961928b2-f95d-4775-88c0-c80ab72bd7be/konnectivity-agent/0.log" limit=1024
Jul 24 13:34:31.756699 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.756257    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hggbw"
Jul 24 13:34:31.882979 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.882867    2527 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qnqt6_961928b2-f95d-4775-88c0-c80ab72bd7be/konnectivity-agent/0.log" limit=1024
Jul 24 13:34:31.911234 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.911201    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf"]
Jul 24 13:34:31.921469 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:31.921413    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b2a17c_563e_4e60_868b_41f51530c308.slice/crio-34e132bb43cabee6ae07a7e5b9366bd66d64d8dc423e5b7e3f4d87e1d71c1589 WatchSource:0}: Error finding container 34e132bb43cabee6ae07a7e5b9366bd66d64d8dc423e5b7e3f4d87e1d71c1589: Status 404 returned error can't find the container with id 34e132bb43cabee6ae07a7e5b9366bd66d64d8dc423e5b7e3f4d87e1d71c1589
Jul 24 13:34:31.934534 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.934501    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd"]
Jul 24 13:34:31.962283 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.962240    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-xqgfh"]
Jul 24 13:34:31.970595 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.970566    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7896g"]
Jul 24 13:34:31.981481 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.981423    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7"]
Jul 24 13:34:31.988023 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:31.987994    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq"]
Jul 24 13:34:32.026643 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.026558    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hggbw"]
Jul 24 13:34:32.033691 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.033663    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:32.033803 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.033702    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:32.033803 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.033776    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:32.033925 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.033805    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:32.033925 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.033826    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:32.033925 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.033852    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.033834031 +0000 UTC m=+37.075881914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:32.033925 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.033891    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:32.033925 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.033897    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.033942    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.033926035 +0000 UTC m=+37.075973906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.033960    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.033950943 +0000 UTC m=+37.075998810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.034006    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.034010    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.034020    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.034045    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.03402991 +0000 UTC m=+37.076077774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:32.034152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.034065    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.034054526 +0000 UTC m=+37.076102394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:32.134724 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.134686    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:32.134894 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.134845    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:32.134981 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:32.134914    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:33.134898994 +0000 UTC m=+37.176946857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:32.511517 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.511481    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:34:32.511704 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.511525    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:34:32.514105 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.514083    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:34:32.514808 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.514151    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvb9z\""
Jul 24 13:34:32.818928 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:32.818676    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf" event={"ID":"65b2a17c-563e-4e60-868b-41f51530c308","Type":"ContainerStarted","Data":"34e132bb43cabee6ae07a7e5b9366bd66d64d8dc423e5b7e3f4d87e1d71c1589"}
Jul 24 13:34:33.040948 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.040909    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:33.041115 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.040961    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:33.041115 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041075    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:33.041115 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.041091    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041149    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.041128255 +0000 UTC m=+39.083176157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041078    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041179    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041190    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.041218    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041230    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.04121428 +0000 UTC m=+39.083262157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:33.041273 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041251    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.041241327 +0000 UTC m=+39.083289193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:33.041592 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041282    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.041270594 +0000 UTC m=+39.083318457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:33.041592 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.041302    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:33.041592 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041423    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:33.041592 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.041476    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.041464604 +0000 UTC m=+39.083512472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:33.142779 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.142749    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:33.142949 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.142899    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:33.143001 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:33.142968    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:35.142949442 +0000 UTC m=+39.184997305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:33.453869 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.453792    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49397988_6b3a_4429_a7c0_b700fd2b1088.slice/crio-92587f3587a48cef4ab4414619b4a34cdea041efd4fc717cb67ccd4a886fb03c WatchSource:0}: Error finding container 92587f3587a48cef4ab4414619b4a34cdea041efd4fc717cb67ccd4a886fb03c: Status 404 returned error can't find the container with id 92587f3587a48cef4ab4414619b4a34cdea041efd4fc717cb67ccd4a886fb03c
Jul 24 13:34:33.454874 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.454465    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1587d831_aefd_4241_8f63_e0a31867f65d.slice/crio-d65a52e6cb9b3c34c1eff305217638d7cc0d51e9a3817a1609a49d4d68b372c9 WatchSource:0}: Error finding container d65a52e6cb9b3c34c1eff305217638d7cc0d51e9a3817a1609a49d4d68b372c9: Status 404 returned error can't find the container with id d65a52e6cb9b3c34c1eff305217638d7cc0d51e9a3817a1609a49d4d68b372c9
Jul 24 13:34:33.455659 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.455577    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d308633_b904_498a_b0b1_f8ca0a66fd0f.slice/crio-a1037a8f80c89db7e2ea71568ab433160a853407843a01649f3d739da66bcc96 WatchSource:0}: Error finding container a1037a8f80c89db7e2ea71568ab433160a853407843a01649f3d739da66bcc96: Status 404 returned error can't find the container with id a1037a8f80c89db7e2ea71568ab433160a853407843a01649f3d739da66bcc96
Jul 24 13:34:33.456399 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.456371    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80506080_1458_4891_ba70_00afb1e62048.slice/crio-ccbe215f7f647130bb56b8543ce5a8e98d0a80669a1f29bb85b853575d8e74cb WatchSource:0}: Error finding container ccbe215f7f647130bb56b8543ce5a8e98d0a80669a1f29bb85b853575d8e74cb: Status 404 returned error can't find the container with id ccbe215f7f647130bb56b8543ce5a8e98d0a80669a1f29bb85b853575d8e74cb
Jul 24 13:34:33.457217 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.457188    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d55935_92b8_48bb_93de_593c41e1c7d3.slice/crio-bd50175acde92d0988bb7c79e5be4334bddd7e0619a7da826d2a1c7f00e1e5d2 WatchSource:0}: Error finding container bd50175acde92d0988bb7c79e5be4334bddd7e0619a7da826d2a1c7f00e1e5d2: Status 404 returned error can't find the container with id bd50175acde92d0988bb7c79e5be4334bddd7e0619a7da826d2a1c7f00e1e5d2
Jul 24 13:34:33.458284 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:33.458253    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8d0646_bb00_4c87_a2c8_4f5ece6de933.slice/crio-f42a569f8960f11f4e02446d6137ef97abc1f1bac18fdfcf9f8aa01964a95140 WatchSource:0}: Error finding container f42a569f8960f11f4e02446d6137ef97abc1f1bac18fdfcf9f8aa01964a95140: Status 404 returned error can't find the container with id f42a569f8960f11f4e02446d6137ef97abc1f1bac18fdfcf9f8aa01964a95140
Jul 24 13:34:33.822486 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.822371    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq" event={"ID":"4b8d0646-bb00-4c87-a2c8-4f5ece6de933","Type":"ContainerStarted","Data":"f42a569f8960f11f4e02446d6137ef97abc1f1bac18fdfcf9f8aa01964a95140"}
Jul 24 13:34:33.824470 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.824419    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" event={"ID":"6d308633-b904-498a-b0b1-f8ca0a66fd0f","Type":"ContainerStarted","Data":"a1037a8f80c89db7e2ea71568ab433160a853407843a01649f3d739da66bcc96"}
Jul 24 13:34:33.825623 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.825589    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hggbw" event={"ID":"49397988-6b3a-4429-a7c0-b700fd2b1088","Type":"ContainerStarted","Data":"92587f3587a48cef4ab4414619b4a34cdea041efd4fc717cb67ccd4a886fb03c"}
Jul 24 13:34:33.826952 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.826926    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd" event={"ID":"80506080-1458-4891-ba70-00afb1e62048","Type":"ContainerStarted","Data":"ccbe215f7f647130bb56b8543ce5a8e98d0a80669a1f29bb85b853575d8e74cb"}
Jul 24 13:34:33.829914 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.829888    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="906259440a403239de79f058cb69a87c593bea64be6e7fa096e0b94456d67417" exitCode=0
Jul 24 13:34:33.830029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.829970    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"906259440a403239de79f058cb69a87c593bea64be6e7fa096e0b94456d67417"}
Jul 24 13:34:33.832028 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.832001    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7" event={"ID":"e7d55935-92b8-48bb-93de-593c41e1c7d3","Type":"ContainerStarted","Data":"bd50175acde92d0988bb7c79e5be4334bddd7e0619a7da826d2a1c7f00e1e5d2"}
Jul 24 13:34:33.833685 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:33.833661    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7896g" event={"ID":"1587d831-aefd-4241-8f63-e0a31867f65d","Type":"ContainerStarted","Data":"d65a52e6cb9b3c34c1eff305217638d7cc0d51e9a3817a1609a49d4d68b372c9"}
Jul 24 13:34:34.845084 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:34.844561    2527 generic.go:356] "Generic (PLEG): container finished" podID="78698a3e-c8e7-473e-87e2-1fb4894d2c59" containerID="d335913b134e723bf5d987e42dfd6e8b13eb3df8339e851dc39c8b6b5ca8bdb3" exitCode=0
Jul 24 13:34:34.845084 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:34.844640    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerDied","Data":"d335913b134e723bf5d987e42dfd6e8b13eb3df8339e851dc39c8b6b5ca8bdb3"}
Jul 24 13:34:35.066503 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.066458    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:35.066722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.066516    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:35.066722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.066544    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:35.066722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.066623    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:35.066722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.066653    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:35.066963 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.066791    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:35.066963 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.066852    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.066833704 +0000 UTC m=+43.108881572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:35.067257 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067212    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:35.067320 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067278    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.067261231 +0000 UTC m=+43.109309099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:35.067320 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067299    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.067289846 +0000 UTC m=+43.109337715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:35.067452 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067330    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:35.067452 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067378    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.067359575 +0000 UTC m=+43.109407438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:35.067452 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067392    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:35.067452 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067403    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:35.067659 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.067457    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.067445989 +0000 UTC m=+43.109493870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:35.168070 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:35.167976    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:35.168400 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.168216    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:35.168400 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:35.168281    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:39.168260526 +0000 UTC m=+43.210308396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:36.493853 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:36.493813    2527 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qnqt6_961928b2-f95d-4775-88c0-c80ab72bd7be/konnectivity-agent/0.log" limit=1024
Jul 24 13:34:36.572724 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:36.572698    2527 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qnqt6_961928b2-f95d-4775-88c0-c80ab72bd7be/konnectivity-agent/0.log" limit=1024
Jul 24 13:34:39.106196 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.106136    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:39.106196 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.106205    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.106288    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106327    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.106304686 +0000 UTC m=+51.148352554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106362    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.106381    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106406    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.106395307 +0000 UTC m=+51.148443170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106456    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.106458    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106473    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106484    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106511    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.106499474 +0000 UTC m=+51.148547337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106526    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.106517766 +0000 UTC m=+51.148565639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106535    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:39.106805 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.106562    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.106550397 +0000 UTC m=+51.148598260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:39.207869 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:39.207831    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:39.208041 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.207993    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:39.208100 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:39.208070    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:47.208050029 +0000 UTC m=+51.250097896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:41.870382 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.870342    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq" event={"ID":"4b8d0646-bb00-4c87-a2c8-4f5ece6de933","Type":"ContainerStarted","Data":"8809bcfa2d8b6fe3e67d00992cc9656f92d48b25770300a428e6640d63eee56c"}
Jul 24 13:34:41.872105 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.872081    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/0.log"
Jul 24 13:34:41.872247 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.872126    2527 generic.go:356] "Generic (PLEG): container finished" podID="6d308633-b904-498a-b0b1-f8ca0a66fd0f" containerID="df26e0b8c58bff2bc50a27d14536565d8726634338e7c2f0fefbd1dc40dcb07b" exitCode=255
Jul 24 13:34:41.872247 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.872226    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" event={"ID":"6d308633-b904-498a-b0b1-f8ca0a66fd0f","Type":"ContainerDied","Data":"df26e0b8c58bff2bc50a27d14536565d8726634338e7c2f0fefbd1dc40dcb07b"}
Jul 24 13:34:41.872390 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.872370    2527 scope.go:117] "RemoveContainer" containerID="df26e0b8c58bff2bc50a27d14536565d8726634338e7c2f0fefbd1dc40dcb07b"
Jul 24 13:34:41.873881 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.873856    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hggbw" event={"ID":"49397988-6b3a-4429-a7c0-b700fd2b1088","Type":"ContainerStarted","Data":"16961b13c667c4b7f18e3d1051ff09ac61a1e1b94cb4d31180d1bba73e59083a"}
Jul 24 13:34:41.875482 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.875459    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd" event={"ID":"80506080-1458-4891-ba70-00afb1e62048","Type":"ContainerStarted","Data":"648317d76c868f72940800e181dfb32eff07e19c25bc3c16101be604588b12b5"}
Jul 24 13:34:41.879569 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.879544    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-szkt4" event={"ID":"78698a3e-c8e7-473e-87e2-1fb4894d2c59","Type":"ContainerStarted","Data":"67359f77e17510e16a942813e8a42de2826402f1bbb40ff63dd918090ae580f0"}
Jul 24 13:34:41.881215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.881028    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf" event={"ID":"65b2a17c-563e-4e60-868b-41f51530c308","Type":"ContainerStarted","Data":"0a27bf04dacc5653ab9e0b1392caac73d4a2445abb60483051c1199a092ebd1d"}
Jul 24 13:34:41.883158 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.883133    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7" event={"ID":"e7d55935-92b8-48bb-93de-593c41e1c7d3","Type":"ContainerStarted","Data":"a841e4e4070023f3188bba3550dd8442935f4d13aab21d6d9c2f3388241be9e1"}
Jul 24 13:34:41.884890 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.884866    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7896g" event={"ID":"1587d831-aefd-4241-8f63-e0a31867f65d","Type":"ContainerStarted","Data":"67e5d6c597e908d22e717130c26601670c00ff059f08ee53526dfa19458340d5"}
Jul 24 13:34:41.888420 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.888370    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-r9pgq" podStartSLOduration=6.407595755 podStartE2EDuration="13.888354719s" totalImagesPullingTime="7.480758964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:41.887540551 +0000 UTC m=+45.929588437" watchObservedRunningTime="2026-07-24 13:34:41.888354719 +0000 UTC m=+45.930402606"
Jul 24 13:34:41.914522 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.914476    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-szkt4" podStartSLOduration=12.279219312 podStartE2EDuration="44.914458684s" totalImagesPullingTime="32.635239372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:41.91308472 +0000 UTC m=+45.955132604" watchObservedRunningTime="2026-07-24 13:34:41.914458684 +0000 UTC m=+45.956506560"
Jul 24 13:34:41.935868 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.935797    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf" podStartSLOduration=4.884915199 podStartE2EDuration="13.935776841s" totalImagesPullingTime="9.050861642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:41.935597749 +0000 UTC m=+45.977645636" watchObservedRunningTime="2026-07-24 13:34:41.935776841 +0000 UTC m=+45.977824724"
Jul 24 13:34:41.963710 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.963636    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-7896g" podStartSLOduration=6.443141049 podStartE2EDuration="13.963612344s" totalImagesPullingTime="7.520471295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:41.9622432 +0000 UTC m=+46.004291085" watchObservedRunningTime="2026-07-24 13:34:41.963612344 +0000 UTC m=+46.005660234"
Jul 24 13:34:41.983925 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:41.983866    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd" podStartSLOduration=6.466087602 podStartE2EDuration="13.983845268s" totalImagesPullingTime="7.517757666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:41.982454686 +0000 UTC m=+46.024502572" watchObservedRunningTime="2026-07-24 13:34:41.983845268 +0000 UTC m=+46.025893154"
Jul 24 13:34:42.023740 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.023688    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5x9s7" podStartSLOduration=6.2718624720000005 podStartE2EDuration="14.023666851s" totalImagesPullingTime="7.751804379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:42.022727861 +0000 UTC m=+46.064775747" watchObservedRunningTime="2026-07-24 13:34:42.023666851 +0000 UTC m=+46.065714730"
Jul 24 13:34:42.046895 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.046797    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hggbw" podStartSLOduration=3.525165309 podStartE2EDuration="11.046774411s" totalImagesPullingTime="7.521609102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:42.045261147 +0000 UTC m=+46.087309043" watchObservedRunningTime="2026-07-24 13:34:42.046774411 +0000 UTC m=+46.088822300"
Jul 24 13:34:42.428010 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.427971    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"]
Jul 24 13:34:42.454630 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.454599    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"]
Jul 24 13:34:42.454777 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.454727    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"
Jul 24 13:34:42.458012 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.457983    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:42.458181 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.457993    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 13:34:42.458525 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.458506    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pk9rl\""
Jul 24 13:34:42.539315 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.539267    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrhg\" (UniqueName: \"kubernetes.io/projected/4407ce96-9260-42e3-aead-f8006c87f3b1-kube-api-access-vrrhg\") pod \"migrator-6865c9967-5cp48\" (UID: \"4407ce96-9260-42e3-aead-f8006c87f3b1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"
Jul 24 13:34:42.567895 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.567853    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rvln4"]
Jul 24 13:34:42.584651 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.584617    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.585871 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.585847    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rvln4"]
Jul 24 13:34:42.588370 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.588349    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnnsn\""
Jul 24 13:34:42.588727 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.588703    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 13:34:42.588832 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.588694    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 13:34:42.640377 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640346    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrhg\" (UniqueName: \"kubernetes.io/projected/4407ce96-9260-42e3-aead-f8006c87f3b1-kube-api-access-vrrhg\") pod \"migrator-6865c9967-5cp48\" (UID: \"4407ce96-9260-42e3-aead-f8006c87f3b1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"
Jul 24 13:34:42.640579 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640401    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.640579 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640483    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de81ac99-198b-421c-88d4-998d005dcb61-data-volume\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.640696 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640608    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.640748 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640703    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de81ac99-198b-421c-88d4-998d005dcb61-crio-socket\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.640821 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.640797    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvzk\" (UniqueName: \"kubernetes.io/projected/de81ac99-198b-421c-88d4-998d005dcb61-kube-api-access-cgvzk\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.663379 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.663345    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrhg\" (UniqueName: \"kubernetes.io/projected/4407ce96-9260-42e3-aead-f8006c87f3b1-kube-api-access-vrrhg\") pod \"migrator-6865c9967-5cp48\" (UID: \"4407ce96-9260-42e3-aead-f8006c87f3b1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"
Jul 24 13:34:42.741669 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741577    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.741843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741669    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de81ac99-198b-421c-88d4-998d005dcb61-crio-socket\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.741843 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:42.741708    2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 13:34:42.741843 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:42.741781    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:43.241764752 +0000 UTC m=+47.283812619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : configmap "kube-rbac-proxy" not found
Jul 24 13:34:42.741843 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741716    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvzk\" (UniqueName: \"kubernetes.io/projected/de81ac99-198b-421c-88d4-998d005dcb61-kube-api-access-cgvzk\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.742030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741903    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.742030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741941    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de81ac99-198b-421c-88d4-998d005dcb61-crio-socket\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.742030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.741963    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de81ac99-198b-421c-88d4-998d005dcb61-data-volume\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.742169 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:42.742085    2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:42.742222 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:42.742171    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:43.242151012 +0000 UTC m=+47.284198877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:42.742342 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.742323    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de81ac99-198b-421c-88d4-998d005dcb61-data-volume\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.751281 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.751253    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvzk\" (UniqueName: \"kubernetes.io/projected/de81ac99-198b-421c-88d4-998d005dcb61-kube-api-access-cgvzk\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:42.766283 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.766252    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"
Jul 24 13:34:42.890610 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.890583    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:34:42.891030 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.891016    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/0.log"
Jul 24 13:34:42.891093 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.891057    2527 generic.go:356] "Generic (PLEG): container finished" podID="6d308633-b904-498a-b0b1-f8ca0a66fd0f" containerID="aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a" exitCode=255
Jul 24 13:34:42.891212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.891175    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" event={"ID":"6d308633-b904-498a-b0b1-f8ca0a66fd0f","Type":"ContainerDied","Data":"aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a"}
Jul 24 13:34:42.891270 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.891256    2527 scope.go:117] "RemoveContainer" containerID="df26e0b8c58bff2bc50a27d14536565d8726634338e7c2f0fefbd1dc40dcb07b"
Jul 24 13:34:42.891365 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.891351    2527 scope.go:117] "RemoveContainer" containerID="aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a"
Jul 24 13:34:42.891626 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:42.891592    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xqgfh_openshift-console-operator(6d308633-b904-498a-b0b1-f8ca0a66fd0f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" podUID="6d308633-b904-498a-b0b1-f8ca0a66fd0f"
Jul 24 13:34:42.936289 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:42.936262    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48"]
Jul 24 13:34:42.939644 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:42.939618    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4407ce96_9260_42e3_aead_f8006c87f3b1.slice/crio-73d996c867a50ff9775cf80256aec763d07a645036780aad1edb9d1154f087b6 WatchSource:0}: Error finding container 73d996c867a50ff9775cf80256aec763d07a645036780aad1edb9d1154f087b6: Status 404 returned error can't find the container with id 73d996c867a50ff9775cf80256aec763d07a645036780aad1edb9d1154f087b6
Jul 24 13:34:43.247093 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:43.247053    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:43.247263 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:43.247188    2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 13:34:43.247263 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:43.247195    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:43.247263 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:43.247259    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:44.247239099 +0000 UTC m=+48.289286980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : configmap "kube-rbac-proxy" not found
Jul 24 13:34:43.247369 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:43.247318    2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:43.247404 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:43.247373    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:44.247359479 +0000 UTC m=+48.289407344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:43.898955 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:43.898907    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48" event={"ID":"4407ce96-9260-42e3-aead-f8006c87f3b1","Type":"ContainerStarted","Data":"73d996c867a50ff9775cf80256aec763d07a645036780aad1edb9d1154f087b6"}
Jul 24 13:34:43.900646 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:43.900622    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:34:43.900992 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:43.900976    2527 scope.go:117] "RemoveContainer" containerID="aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a"
Jul 24 13:34:43.901186 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:43.901160    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xqgfh_openshift-console-operator(6d308633-b904-498a-b0b1-f8ca0a66fd0f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" podUID="6d308633-b904-498a-b0b1-f8ca0a66fd0f"
Jul 24 13:34:44.256241 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.256162    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:44.256418 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.256298    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:44.256418 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:44.256314    2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 13:34:44.256418 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:44.256397    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:46.256374315 +0000 UTC m=+50.298422179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : configmap "kube-rbac-proxy" not found
Jul 24 13:34:44.256623 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:44.256483    2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:44.256623 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:44.256551    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:46.256532737 +0000 UTC m=+50.298580621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:44.761238 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.761215    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z26h4_a1e62ee6-1992-460f-ab28-59e9ceb5d89c/dns-node-resolver/0.log"
Jul 24 13:34:44.905507 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.905463    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48" event={"ID":"4407ce96-9260-42e3-aead-f8006c87f3b1","Type":"ContainerStarted","Data":"f4eeab901a96f5244595925f0a6444c246dfd6bedfab41d00038370b985bcf57"}
Jul 24 13:34:44.905852 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.905560    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48" event={"ID":"4407ce96-9260-42e3-aead-f8006c87f3b1","Type":"ContainerStarted","Data":"27d0819680ab8ba4a7454ad264fbc7fe41f4ac3cb0adc2a40f25ca95a87df4bf"}
Jul 24 13:34:44.924424 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:44.924330    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-5cp48" podStartSLOduration=1.202979184 podStartE2EDuration="2.924315611s" totalImagesPullingTime="1.721336427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:44.922642556 +0000 UTC m=+48.964690451" watchObservedRunningTime="2026-07-24 13:34:44.924315611 +0000 UTC m=+48.966363496"
Jul 24 13:34:45.133682 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.133644    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hzpmg"]
Jul 24 13:34:45.169818 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.169782    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hzpmg"]
Jul 24 13:34:45.169988 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.169898    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.172246 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.172220    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 13:34:45.172397 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.172336    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n5lws\""
Jul 24 13:34:45.172607 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.172593    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 13:34:45.172675 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.172661    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 13:34:45.172726 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.172695    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 13:34:45.268525 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.268483    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-key\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.268718 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.268562    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-cabundle\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.268718 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.268661    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5j5\" (UniqueName: \"kubernetes.io/projected/cc564807-d60a-4467-a3fb-4d1b9711112b-kube-api-access-kc5j5\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.369808 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.369771    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-cabundle\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.369995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.369853    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5j5\" (UniqueName: \"kubernetes.io/projected/cc564807-d60a-4467-a3fb-4d1b9711112b-kube-api-access-kc5j5\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.369995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.369955    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-key\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.370578 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.370555    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-cabundle\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.372493 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.372471    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cc564807-d60a-4467-a3fb-4d1b9711112b-signing-key\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.378804 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.378781    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5j5\" (UniqueName: \"kubernetes.io/projected/cc564807-d60a-4467-a3fb-4d1b9711112b-kube-api-access-kc5j5\") pod \"service-ca-658d7b7c78-hzpmg\" (UID: \"cc564807-d60a-4467-a3fb-4d1b9711112b\") " pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.478599 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.478513    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg"
Jul 24 13:34:45.561922 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.561899    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nhlx_8705eba6-6f23-4ad9-8a91-5cdee758f61a/node-ca/0.log"
Jul 24 13:34:45.639811 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.639783    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-hzpmg"]
Jul 24 13:34:45.643022 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:45.642993    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc564807_d60a_4467_a3fb_4d1b9711112b.slice/crio-bdcc036d98cdb58646b9821b7889f378290fed8ad26a0cee154fe9a046aca800 WatchSource:0}: Error finding container bdcc036d98cdb58646b9821b7889f378290fed8ad26a0cee154fe9a046aca800: Status 404 returned error can't find the container with id bdcc036d98cdb58646b9821b7889f378290fed8ad26a0cee154fe9a046aca800
Jul 24 13:34:45.910662 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.910624    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg" event={"ID":"cc564807-d60a-4467-a3fb-4d1b9711112b","Type":"ContainerStarted","Data":"b500f150e8bcaadf9d0e1cfcba2a404bae7e65fb5bd4fca1efb9f0effbfec35e"}
Jul 24 13:34:45.911047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.910676    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg" event={"ID":"cc564807-d60a-4467-a3fb-4d1b9711112b","Type":"ContainerStarted","Data":"bdcc036d98cdb58646b9821b7889f378290fed8ad26a0cee154fe9a046aca800"}
Jul 24 13:34:45.938234 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:45.938188    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-hzpmg" podStartSLOduration=0.938155839 podStartE2EDuration="938.155839ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:45.937482587 +0000 UTC m=+49.979530474" watchObservedRunningTime="2026-07-24 13:34:45.938155839 +0000 UTC m=+49.980203724"
Jul 24 13:34:46.166418 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.166384    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hggbw_49397988-6b3a-4429-a7c0-b700fd2b1088/serve-healthcheck-canary/0.log"
Jul 24 13:34:46.277602 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.277567    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:46.277764 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.277680    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:46.277764 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:46.277708    2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 13:34:46.277838 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:46.277784    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:50.277762695 +0000 UTC m=+54.319810595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : configmap "kube-rbac-proxy" not found
Jul 24 13:34:46.277838 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:46.277828    2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:46.277984 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:46.277899    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:50.277885831 +0000 UTC m=+54.319933697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:46.362563 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.362531    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-5cp48_4407ce96-9260-42e3-aead-f8006c87f3b1/migrator/0.log"
Jul 24 13:34:46.561721 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.561643    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-5cp48_4407ce96-9260-42e3-aead-f8006c87f3b1/graceful-termination/0.log"
Jul 24 13:34:46.769566 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:46.769533    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf_65b2a17c-563e-4e60-868b-41f51530c308/kube-storage-version-migrator-operator/0.log"
Jul 24 13:34:47.184818 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.184778    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.184863    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.184906    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.184931    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.184949    2527 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185021    2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185047    2527 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185056    2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185023    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls podName:8c235d3d-e3cf-47ba-bc21-baf3f442e5b8 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.185001861 +0000 UTC m=+67.227049728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-k8tsd" (UID: "8c235d3d-e3cf-47ba-bc21-baf3f442e5b8") : secret "cluster-monitoring-operator-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185067    2527 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678d47fbf-87pvp: secret "image-registry-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185095    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls podName:c02748a9-e0c5-4d09-93e2-5d65dd400703 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.185078271 +0000 UTC m=+67.227126140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls") pod "cluster-samples-operator-6748489656-4blgh" (UID: "c02748a9-e0c5-4d09-93e2-5d65dd400703") : secret "samples-operator-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185121    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls podName:6ed725e5-f396-4aaf-beed-dee1fc3b57b6 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.185112289 +0000 UTC m=+67.227160156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls") pod "image-registry-678d47fbf-87pvp" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6") : secret "image-registry-tls" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185139    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.185130335 +0000 UTC m=+67.227178200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : secret "router-metrics-certs-default" not found
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.185153    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:34:47.185422 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.185251    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle podName:81c14746-f5a8-4bb3-8daa-93a2b84a3e6f nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.185240971 +0000 UTC m=+67.227288847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle") pod "router-default-d4b8fd45f-z8bhm" (UID: "81c14746-f5a8-4bb3-8daa-93a2b84a3e6f") : configmap references non-existent config key: service-ca.crt
Jul 24 13:34:47.286359 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:47.286311    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:34:47.286581 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.286518    2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 13:34:47.286645 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:47.286622    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls podName:1a6d4f0a-0a6d-479f-a397-2eb4b324a169 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:03.28659939 +0000 UTC m=+67.328647257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls") pod "dns-default-c946z" (UID: "1a6d4f0a-0a6d-479f-a397-2eb4b324a169") : secret "dns-default-metrics-tls" not found
Jul 24 13:34:50.315621 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:50.315580    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:50.316149 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:50.315670    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:50.316149 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:50.315747    2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:50.316149 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:50.315820    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls podName:de81ac99-198b-421c-88d4-998d005dcb61 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:58.315800058 +0000 UTC m=+62.357847925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvln4" (UID: "de81ac99-198b-421c-88d4-998d005dcb61") : secret "insights-runtime-extractor-tls" not found
Jul 24 13:34:50.316262 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:50.316151    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de81ac99-198b-421c-88d4-998d005dcb61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:51.676545 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:51.676487    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:51.676932 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:51.676576    2527 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:34:51.676979 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:51.676949    2527 scope.go:117] "RemoveContainer" containerID="aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a"
Jul 24 13:34:51.677126 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:34:51.677106    2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-xqgfh_openshift-console-operator(6d308633-b904-498a-b0b1-f8ca0a66fd0f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" podUID="6d308633-b904-498a-b0b1-f8ca0a66fd0f"
Jul 24 13:34:57.814864 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:57.814830    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8n58"
Jul 24 13:34:58.389300 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.389259    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:58.391711 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.391681    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de81ac99-198b-421c-88d4-998d005dcb61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvln4\" (UID: \"de81ac99-198b-421c-88d4-998d005dcb61\") " pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:58.500960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.500928    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnnsn\""
Jul 24 13:34:58.504714 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.504696    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rvln4"
Jul 24 13:34:58.655821 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.655740    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rvln4"]
Jul 24 13:34:58.660713 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:34:58.660679    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde81ac99_198b_421c_88d4_998d005dcb61.slice/crio-c247086a95ea7ee4ef487fcdc3f2e972a0053a2580281cfe8427161d08eb3a60 WatchSource:0}: Error finding container c247086a95ea7ee4ef487fcdc3f2e972a0053a2580281cfe8427161d08eb3a60: Status 404 returned error can't find the container with id c247086a95ea7ee4ef487fcdc3f2e972a0053a2580281cfe8427161d08eb3a60
Jul 24 13:34:58.957126 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:34:58.957036    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvln4" event={"ID":"de81ac99-198b-421c-88d4-998d005dcb61","Type":"ContainerStarted","Data":"c247086a95ea7ee4ef487fcdc3f2e972a0053a2580281cfe8427161d08eb3a60"}
Jul 24 13:35:01.115149 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.114929    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:35:01.117235 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.117206    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:35:01.128616 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.128590    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b318dd6-b05b-4f18-abc7-6af9858c0690-metrics-certs\") pod \"network-metrics-daemon-gdskt\" (UID: \"8b318dd6-b05b-4f18-abc7-6af9858c0690\") " pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:35:01.216208 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.216166    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:35:01.216398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.216293    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:35:01.219356 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.219326    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzlr\" (UniqueName: \"kubernetes.io/projected/33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6-kube-api-access-gqzlr\") pod \"network-check-target-rj5bn\" (UID: \"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6\") " pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:35:01.219541 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.219383    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:35:01.221090 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.221065    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8g8q\""
Jul 24 13:35:01.229196 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.229167    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c72de-350d-4005-9d29-bf40ae47ac5c-original-pull-secret\") pod \"global-pull-secret-syncer-qq8rj\" (UID: \"ac3c72de-350d-4005-9d29-bf40ae47ac5c\") " pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:35:01.229332 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.229237    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gdskt"
Jul 24 13:35:01.326426 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.326383    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvb9z\""
Jul 24 13:35:01.331250 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.331223    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qq8rj"
Jul 24 13:35:01.335622 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:01.335599    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:35:02.508108 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:02.508075    2527 scope.go:117] "RemoveContainer" containerID="aa40ddc5cfac9f9d5cca0499d1973336b877541e68140222866b22629c87a71a"
Jul 24 13:35:03.236201 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.236161    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:03.236398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.236211    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:35:03.236398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.236280    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:35:03.236398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.236311    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:03.236398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.236329    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:03.238060 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.237537    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-service-ca-bundle\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:03.240233 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.240151    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c02748a9-e0c5-4d09-93e2-5d65dd400703-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4blgh\" (UID: \"c02748a9-e0c5-4d09-93e2-5d65dd400703\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:35:03.240561 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.240537    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"image-registry-678d47fbf-87pvp\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") " pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:03.243840 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.243811    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81c14746-f5a8-4bb3-8daa-93a2b84a3e6f-metrics-certs\") pod \"router-default-d4b8fd45f-z8bhm\" (UID: \"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f\") " pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:03.244693 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.244668    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c235d3d-e3cf-47ba-bc21-baf3f442e5b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-k8tsd\" (UID: \"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:35:03.337360 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.337318    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:35:03.339995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.339968    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a6d4f0a-0a6d-479f-a397-2eb4b324a169-metrics-tls\") pod \"dns-default-c946z\" (UID: \"1a6d4f0a-0a6d-479f-a397-2eb4b324a169\") " pod="openshift-dns/dns-default-c946z"
Jul 24 13:35:03.420105 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.420019    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hklft\""
Jul 24 13:35:03.427860 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.427826    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:03.429304 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.429271    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x2wfq\""
Jul 24 13:35:03.437128 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.437088    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"
Jul 24 13:35:03.447716 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.447678    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jd4l2\""
Jul 24 13:35:03.456697 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.456659    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:03.463605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.463585    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dvmkc\""
Jul 24 13:35:03.472287 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.472256    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"
Jul 24 13:35:03.561942 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.561905    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kzzx2\""
Jul 24 13:35:03.570147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.569815    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c946z"
Jul 24 13:35:03.859058 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.859024    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh"]
Jul 24 13:35:03.881838 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.881805    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d4b8fd45f-z8bhm"]
Jul 24 13:35:03.887669 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:03.887631    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c14746_f5a8_4bb3_8daa_93a2b84a3e6f.slice/crio-51d763cd6826dfa06e5bcf4fa2b38d69d82d05273235db16f44d69af193c0acf WatchSource:0}: Error finding container 51d763cd6826dfa06e5bcf4fa2b38d69d82d05273235db16f44d69af193c0acf: Status 404 returned error can't find the container with id 51d763cd6826dfa06e5bcf4fa2b38d69d82d05273235db16f44d69af193c0acf
Jul 24 13:35:03.911487 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.911464    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c946z"]
Jul 24 13:35:03.916700 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:03.916675    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6d4f0a_0a6d_479f_a397_2eb4b324a169.slice/crio-72e621b965ab299fca66f644b5146fa57df1108ea4f93077243be4b10887af85 WatchSource:0}: Error finding container 72e621b965ab299fca66f644b5146fa57df1108ea4f93077243be4b10887af85: Status 404 returned error can't find the container with id 72e621b965ab299fca66f644b5146fa57df1108ea4f93077243be4b10887af85
Jul 24 13:35:03.958160 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.958131    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rj5bn"]
Jul 24 13:35:03.961059 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:03.961025    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d358ee_85cd_41c8_9f37_0e7d7a0e8ac6.slice/crio-838e864fa9b0a18279db890e984cedb986b1a24fa7dad9f71445c113c1cea582 WatchSource:0}: Error finding container 838e864fa9b0a18279db890e984cedb986b1a24fa7dad9f71445c113c1cea582: Status 404 returned error can't find the container with id 838e864fa9b0a18279db890e984cedb986b1a24fa7dad9f71445c113c1cea582
Jul 24 13:35:03.977532 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.977472    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:35:03.977664 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.977576    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" event={"ID":"6d308633-b904-498a-b0b1-f8ca0a66fd0f","Type":"ContainerStarted","Data":"1df4bf794c28ef282b06879d5e1bf50362b15efb4e0e28119599a03e5b04b7f5"}
Jul 24 13:35:03.978840 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.978800    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm" event={"ID":"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f","Type":"ContainerStarted","Data":"51d763cd6826dfa06e5bcf4fa2b38d69d82d05273235db16f44d69af193c0acf"}
Jul 24 13:35:03.980048 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.980023    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh" event={"ID":"c02748a9-e0c5-4d09-93e2-5d65dd400703","Type":"ContainerStarted","Data":"eb58cd2aba1c9ab17f7177ee7a83d42888353f317564a8ccff1731ccf3ae5a87"}
Jul 24 13:35:03.981230 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.981131    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rj5bn" event={"ID":"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6","Type":"ContainerStarted","Data":"838e864fa9b0a18279db890e984cedb986b1a24fa7dad9f71445c113c1cea582"}
Jul 24 13:35:03.982283 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.982261    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c946z" event={"ID":"1a6d4f0a-0a6d-479f-a397-2eb4b324a169","Type":"ContainerStarted","Data":"72e621b965ab299fca66f644b5146fa57df1108ea4f93077243be4b10887af85"}
Jul 24 13:35:03.989266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:03.989098    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:35:04.010571 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.010515    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh" podStartSLOduration=28.483995614 podStartE2EDuration="36.010493972s" totalImagesPullingTime="7.526498358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:04.008911081 +0000 UTC m=+68.050958967" watchObservedRunningTime="2026-07-24 13:35:04.010493972 +0000 UTC m=+68.052541859"
Jul 24 13:35:04.026924 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.026885    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qq8rj"]
Jul 24 13:35:04.030656 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.030494    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gdskt"]
Jul 24 13:35:04.032310 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.032267    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd"]
Jul 24 13:35:04.033918 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:04.033887    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3c72de_350d_4005_9d29_bf40ae47ac5c.slice/crio-124aa7de137f09e106c74934502d9f063d260d1dc0aed0386aceb711f031fefb WatchSource:0}: Error finding container 124aa7de137f09e106c74934502d9f063d260d1dc0aed0386aceb711f031fefb: Status 404 returned error can't find the container with id 124aa7de137f09e106c74934502d9f063d260d1dc0aed0386aceb711f031fefb
Jul 24 13:35:04.035194 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:04.035149    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b318dd6_b05b_4f18_abc7_6af9858c0690.slice/crio-bf815d4420b7993c212f9626ce93ea51e2f064972b51b15862cd17f368b23c78 WatchSource:0}: Error finding container bf815d4420b7993c212f9626ce93ea51e2f064972b51b15862cd17f368b23c78: Status 404 returned error can't find the container with id bf815d4420b7993c212f9626ce93ea51e2f064972b51b15862cd17f368b23c78
Jul 24 13:35:04.036225 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:04.036180    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c235d3d_e3cf_47ba_bc21_baf3f442e5b8.slice/crio-98f0375e072933806afa87358084f8a40f794098464a3b7f39db18d7b513fb7f WatchSource:0}: Error finding container 98f0375e072933806afa87358084f8a40f794098464a3b7f39db18d7b513fb7f: Status 404 returned error can't find the container with id 98f0375e072933806afa87358084f8a40f794098464a3b7f39db18d7b513fb7f
Jul 24 13:35:04.044666 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.044641    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-xqgfh"
Jul 24 13:35:04.089646 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.089604    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:35:04.989236 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.989199    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" event={"ID":"6ed725e5-f396-4aaf-beed-dee1fc3b57b6","Type":"ContainerStarted","Data":"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"}
Jul 24 13:35:04.989759 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.989260    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" event={"ID":"6ed725e5-f396-4aaf-beed-dee1fc3b57b6","Type":"ContainerStarted","Data":"7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765"}
Jul 24 13:35:04.989759 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.989303    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:04.991218 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.991189    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm" event={"ID":"81c14746-f5a8-4bb3-8daa-93a2b84a3e6f","Type":"ContainerStarted","Data":"1f4d1c11f72eefa1d6d5de28b88f7c6982ce4badc6a1b351a3f212b8b309ecea"}
Jul 24 13:35:04.993105 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.993072    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvln4" event={"ID":"de81ac99-198b-421c-88d4-998d005dcb61","Type":"ContainerStarted","Data":"7adf559553170217212eca43b432f82b8c34990ad038de6529595fe83f311cfc"}
Jul 24 13:35:04.994398 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.994373    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd" event={"ID":"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8","Type":"ContainerStarted","Data":"98f0375e072933806afa87358084f8a40f794098464a3b7f39db18d7b513fb7f"}
Jul 24 13:35:04.996453 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.996411    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rj5bn" event={"ID":"33d358ee-85cd-41c8-9f37-0e7d7a0e8ac6","Type":"ContainerStarted","Data":"a52b880a0a07d70384e9c3fc61f09456bfc9965c450cc46796fac0cf976e4dbe"}
Jul 24 13:35:04.996604 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.996587    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:35:04.998338 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:04.998283    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdskt" event={"ID":"8b318dd6-b05b-4f18-abc7-6af9858c0690","Type":"ContainerStarted","Data":"bf815d4420b7993c212f9626ce93ea51e2f064972b51b15862cd17f368b23c78"}
Jul 24 13:35:05.000217 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.000192    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qq8rj" event={"ID":"ac3c72de-350d-4005-9d29-bf40ae47ac5c","Type":"ContainerStarted","Data":"124aa7de137f09e106c74934502d9f063d260d1dc0aed0386aceb711f031fefb"}
Jul 24 13:35:05.012837 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.012791    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" podStartSLOduration=68.012760769 podStartE2EDuration="1m8.012760769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:05.01107048 +0000 UTC m=+69.053118366" watchObservedRunningTime="2026-07-24 13:35:05.012760769 +0000 UTC m=+69.054808632"
Jul 24 13:35:05.060267 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.059917    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm" podStartSLOduration=37.05989552 podStartE2EDuration="37.05989552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:05.040602572 +0000 UTC m=+69.082650458" watchObservedRunningTime="2026-07-24 13:35:05.05989552 +0000 UTC m=+69.101943386"
Jul 24 13:35:05.061309 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.061262    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rj5bn" podStartSLOduration=68.061246285 podStartE2EDuration="1m8.061246285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:05.059227679 +0000 UTC m=+69.101275566" watchObservedRunningTime="2026-07-24 13:35:05.061246285 +0000 UTC m=+69.103294171"
Jul 24 13:35:05.457413 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.457363    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:05.460256 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:05.460228    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:06.005863 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:06.005806    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:06.007723 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:06.007700    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d4b8fd45f-z8bhm"
Jul 24 13:35:07.068224 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.068185    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"]
Jul 24 13:35:07.075556 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.075513    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"]
Jul 24 13:35:07.075810 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.075676    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.081515 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081358    2527 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" podUID="fd809ac8-485e-4736-ab74-cd3ffd2f3f41" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.081515 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081475    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Jul 24 13:35:07.081752 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081564    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Jul 24 13:35:07.081752 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081617    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Jul 24 13:35:07.081752 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081664    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret"
Jul 24 13:35:07.081752 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081711    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 24 13:35:07.081984 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081838    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 24 13:35:07.081984 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:07.081907    2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Jul 24 13:35:07.082090 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.082077    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"]
Jul 24 13:35:07.083422 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.083398    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.085544 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.085522    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.088182 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.088154    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 13:35:07.088316 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.088213    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8xsrm\""
Jul 24 13:35:07.088316 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.088224    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 13:35:07.097675 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.097642    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"]
Jul 24 13:35:07.099202 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.099158    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"]
Jul 24 13:35:07.106667 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.106636    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"]
Jul 24 13:35:07.106835 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.106682    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"]
Jul 24 13:35:07.106835 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.106745    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.111269 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.111242    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 13:35:07.111587 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.111568    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bmm6c\""
Jul 24 13:35:07.115066 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.115039    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 13:35:07.123951 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.123911    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"]
Jul 24 13:35:07.164986 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.164951    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-t6vhk"]
Jul 24 13:35:07.168614 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.168577    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:07.171490 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.171466    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-k4fsl\""
Jul 24 13:35:07.172043 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.171726    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 13:35:07.172043 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.171880    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 13:35:07.172617 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.172589    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c18a60-b9cf-4abc-9962-5e35020d8adc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.172802 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.172628    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.172802 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.172705    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrqm\" (UniqueName: \"kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.172802 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.172732    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.172971 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.172930    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-additional-service-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.173066 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173045    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-tmp\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.173136 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173121    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.173234 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173211    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.173290 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173247    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98zv\" (UniqueName: \"kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.173349 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173320    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blkg\" (UniqueName: \"kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.173406 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.173356    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.187693 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.187661    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-t6vhk"]
Jul 24 13:35:07.274756 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274715    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-additional-service-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.274756 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274764    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-tmp\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274795    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274836    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274868    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274902    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2h2\" (UniqueName: \"kubernetes.io/projected/a2a0e067-907f-4aba-9145-5fc654769f71-kube-api-access-mg2h2\") pod \"downloads-7c97dc4555-t6vhk\" (UID: \"a2a0e067-907f-4aba-9145-5fc654769f71\") " pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274942    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.274982 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274964    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98zv\" (UniqueName: \"kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.275165 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.274989    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blkg\" (UniqueName: \"kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.275165 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275008    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.275227 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275154    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-tmp\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.275227 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275212    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c18a60-b9cf-4abc-9962-5e35020d8adc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.275303 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275241    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.275303 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275283    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrqm\" (UniqueName: \"kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.275415 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.275311    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.278244 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.278215    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:07.278388 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.278356    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6c18a60-b9cf-4abc-9962-5e35020d8adc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:07.376243 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.376207    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.376462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.376276    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.376462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.376309    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2h2\" (UniqueName: \"kubernetes.io/projected/a2a0e067-907f-4aba-9145-5fc654769f71-kube-api-access-mg2h2\") pod \"downloads-7c97dc4555-t6vhk\" (UID: \"a2a0e067-907f-4aba-9145-5fc654769f71\") " pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:07.377391 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.377362    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.380792 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.380732    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/964e98b6-6bfe-49cb-8e46-e64cc75ad9ae-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-74xr7\" (UID: \"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.393617 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.393548    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2h2\" (UniqueName: \"kubernetes.io/projected/a2a0e067-907f-4aba-9145-5fc654769f71-kube-api-access-mg2h2\") pod \"downloads-7c97dc4555-t6vhk\" (UID: \"a2a0e067-907f-4aba-9145-5fc654769f71\") " pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:07.449302 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.449266    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"
Jul 24 13:35:07.481148 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.481110    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:07.919585 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.919547    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 13:35:07.926251 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.926217    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-additional-service-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:07.970189 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.970154    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 13:35:07.978926 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:07.978897    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-ca\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.238062 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.237989    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 13:35:08.248419 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.248394    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.275418 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.275384    2527 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert: failed to sync secret cache: timed out waiting for the condition
Jul 24 13:35:08.275629 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.275520    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub podName:fd809ac8-485e-4736-ab74-cd3ffd2f3f41 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:08.775494716 +0000 UTC m=+72.817542579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub" (UniqueName: "kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub") pod "cluster-proxy-proxy-agent-db887f84b-hrvn8" (UID: "fd809ac8-485e-4736-ab74-cd3ffd2f3f41") : failed to sync secret cache: timed out waiting for the condition
Jul 24 13:35:08.275756 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.275721    2527 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-service-proxy-server-certificates: failed to sync secret cache: timed out waiting for the condition
Jul 24 13:35:08.275894 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.275809    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert podName:fd809ac8-485e-4736-ab74-cd3ffd2f3f41 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:08.775791234 +0000 UTC m=+72.817839105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-proxy-server-cert" (UniqueName: "kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert") pod "cluster-proxy-proxy-agent-db887f84b-hrvn8" (UID: "fd809ac8-485e-4736-ab74-cd3ffd2f3f41") : failed to sync secret cache: timed out waiting for the condition
Jul 24 13:35:08.288783 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288751    2527 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.288917 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288802    2527 projected.go:196] Error preparing data for projected volume kube-api-access-5blkg for pod open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.288917 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288764    2527 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.288917 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288858    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg podName:fd809ac8-485e-4736-ab74-cd3ffd2f3f41 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:08.788841439 +0000 UTC m=+72.830889302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5blkg" (UniqueName: "kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg") pod "cluster-proxy-proxy-agent-db887f84b-hrvn8" (UID: "fd809ac8-485e-4736-ab74-cd3ffd2f3f41") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.288917 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288879    2527 projected.go:196] Error preparing data for projected volume kube-api-access-xsrqm for pod open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.289081 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.288929    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm podName:d6c18a60-b9cf-4abc-9962-5e35020d8adc nodeName:}" failed. No retries permitted until 2026-07-24 13:35:08.788912962 +0000 UTC m=+72.830960840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xsrqm" (UniqueName: "kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm") pod "managed-serviceaccount-addon-agent-77dfd5764d-bs5v4" (UID: "d6c18a60-b9cf-4abc-9962-5e35020d8adc") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.291581 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.291557    2527 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.291714 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.291591    2527 projected.go:196] Error preparing data for projected volume kube-api-access-l98zv for pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x: failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.291714 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:08.291645    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv podName:0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:08.79163073 +0000 UTC m=+72.833678592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l98zv" (UniqueName: "kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv") pod "klusterlet-addon-workmgr-7f85dd99f-mhg8x" (UID: "0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 13:35:08.329686 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.329654    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 13:35:08.335210 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.335178    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 13:35:08.454147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.454094    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 13:35:08.526474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.526375    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 13:35:08.790909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.790821    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blkg\" (UniqueName: \"kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.790909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.790879    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.791154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.790916    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrqm\" (UniqueName: \"kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:08.791154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.790968    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.794023 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.793996    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.794165 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.794108    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-hub\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.794354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.794314    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrqm\" (UniqueName: \"kubernetes.io/projected/d6c18a60-b9cf-4abc-9962-5e35020d8adc-kube-api-access-xsrqm\") pod \"managed-serviceaccount-addon-agent-77dfd5764d-bs5v4\" (UID: \"d6c18a60-b9cf-4abc-9962-5e35020d8adc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:08.794845 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.794826    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blkg\" (UniqueName: \"kubernetes.io/projected/fd809ac8-485e-4736-ab74-cd3ffd2f3f41-kube-api-access-5blkg\") pod \"cluster-proxy-proxy-agent-db887f84b-hrvn8\" (UID: \"fd809ac8-485e-4736-ab74-cd3ffd2f3f41\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.892334 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.892288    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98zv\" (UniqueName: \"kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:08.895063 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.895034    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98zv\" (UniqueName: \"kubernetes.io/projected/0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9-kube-api-access-l98zv\") pod \"klusterlet-addon-workmgr-7f85dd99f-mhg8x\" (UID: \"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:08.903909 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.903877    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"
Jul 24 13:35:08.915068 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.915037    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"
Jul 24 13:35:08.941194 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:08.941162    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:11.453255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:11.453220    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8"]
Jul 24 13:35:11.512137 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:11.512101    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd809ac8_485e_4736_ab74_cd3ffd2f3f41.slice/crio-67086584e0f466689b727c01d03e6a1c52806d783f9a948782d45edd41a2555a WatchSource:0}: Error finding container 67086584e0f466689b727c01d03e6a1c52806d783f9a948782d45edd41a2555a: Status 404 returned error can't find the container with id 67086584e0f466689b727c01d03e6a1c52806d783f9a948782d45edd41a2555a
Jul 24 13:35:11.724448 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:11.724389    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"]
Jul 24 13:35:11.745735 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:11.745701    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-t6vhk"]
Jul 24 13:35:11.750154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:11.749865    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4"]
Jul 24 13:35:11.778921 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:11.778350    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-74xr7"]
Jul 24 13:35:11.883111 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:11.882500    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e78c29d_a32a_4c0a_a2af_1d4dab69fbb9.slice/crio-2c22eb03961aed9aa919222179128332783f9212bd26710abbccdecb026e6c9d WatchSource:0}: Error finding container 2c22eb03961aed9aa919222179128332783f9212bd26710abbccdecb026e6c9d: Status 404 returned error can't find the container with id 2c22eb03961aed9aa919222179128332783f9212bd26710abbccdecb026e6c9d
Jul 24 13:35:12.034048 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.034007    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8" event={"ID":"fd809ac8-485e-4736-ab74-cd3ffd2f3f41","Type":"ContainerStarted","Data":"67086584e0f466689b727c01d03e6a1c52806d783f9a948782d45edd41a2555a"}
Jul 24 13:35:12.035494 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.035402    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4" event={"ID":"d6c18a60-b9cf-4abc-9962-5e35020d8adc","Type":"ContainerStarted","Data":"d8cc8387d80e9d3e2f2e43bccc3a4f743ba89bc294fd680ea090187fb0d75f5a"}
Jul 24 13:35:12.037412 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.037385    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh" event={"ID":"c02748a9-e0c5-4d09-93e2-5d65dd400703","Type":"ContainerStarted","Data":"a8bd3d2c57692ddbaf9759601387574859e9eb155a166630a7a734f79d0e7ae6"}
Jul 24 13:35:12.038667 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.038641    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdskt" event={"ID":"8b318dd6-b05b-4f18-abc7-6af9858c0690","Type":"ContainerStarted","Data":"81fae9e6f8efb92c8c51d616471ca516ba80c7ae1c7659509f7c33d41ed525b8"}
Jul 24 13:35:12.039806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.039784    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x" event={"ID":"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9","Type":"ContainerStarted","Data":"2c22eb03961aed9aa919222179128332783f9212bd26710abbccdecb026e6c9d"}
Jul 24 13:35:12.041155 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.041133    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qq8rj" event={"ID":"ac3c72de-350d-4005-9d29-bf40ae47ac5c","Type":"ContainerStarted","Data":"75d5716941c08b9d6c00d740e4114bf41ff13727459dc4ede9f08d48a6835fb2"}
Jul 24 13:35:12.042642 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.042620    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7" event={"ID":"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae","Type":"ContainerStarted","Data":"d479e483b87f90ed9ec5d81ff14310a2a93b3f4e577f298e49071396ed4c7edd"}
Jul 24 13:35:12.043657 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.043638    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-t6vhk" event={"ID":"a2a0e067-907f-4aba-9145-5fc654769f71","Type":"ContainerStarted","Data":"03e35512da71220da0b9e717bc1b0bda953e120d4ed46b1ef12c94d26672ebaf"}
Jul 24 13:35:12.044888 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.044868    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c946z" event={"ID":"1a6d4f0a-0a6d-479f-a397-2eb4b324a169","Type":"ContainerStarted","Data":"9eaac05fcd71bd91f78c60287ea70c6fc3452c47af7887aa0c6e99db0f03998e"}
Jul 24 13:35:12.046184 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.046166    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd" event={"ID":"8c235d3d-e3cf-47ba-bc21-baf3f442e5b8","Type":"ContainerStarted","Data":"2aacbf3b7ed3fed81fbddb71cd707eb67110febdb17d63f28234e51b02dd697b"}
Jul 24 13:35:12.086516 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.086407    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qq8rj" podStartSLOduration=67.607865647 podStartE2EDuration="1m15.086384758s" totalImagesPullingTime="7.478519111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:12.084335459 +0000 UTC m=+76.126383357" watchObservedRunningTime="2026-07-24 13:35:12.086384758 +0000 UTC m=+76.128432647"
Jul 24 13:35:12.133092 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:12.132977    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-k8tsd" podStartSLOduration=37.559211519 podStartE2EDuration="44.132954023s" totalImagesPullingTime="6.573742504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:12.132026715 +0000 UTC m=+76.174074602" watchObservedRunningTime="2026-07-24 13:35:12.132954023 +0000 UTC m=+76.175001911"
Jul 24 13:35:13.061637 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.061551    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh" event={"ID":"c02748a9-e0c5-4d09-93e2-5d65dd400703","Type":"ContainerStarted","Data":"739d3db3b10c5bdd0cf61502cd0ec0044bb4d60f44bc481eff1709fbc4172a8a"}
Jul 24 13:35:13.071294 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.070667    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gdskt" event={"ID":"8b318dd6-b05b-4f18-abc7-6af9858c0690","Type":"ContainerStarted","Data":"8d2aa84718a3134320c4ae410832470136e244744517575977cb0d1210c9056e"}
Jul 24 13:35:13.084329 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.084217    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvln4" event={"ID":"de81ac99-198b-421c-88d4-998d005dcb61","Type":"ContainerStarted","Data":"55f06d2114706d1f0c4049353e86931ed3b67f0b3415835195aee43911af1c5b"}
Jul 24 13:35:13.103089 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.103050    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c946z" event={"ID":"1a6d4f0a-0a6d-479f-a397-2eb4b324a169","Type":"ContainerStarted","Data":"ce733e0f3023c41284dc6891818d534100283a3460d8de36ac13b7c6aef3797f"}
Jul 24 13:35:13.107574 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.107533    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c946z"
Jul 24 13:35:13.108502 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.107922    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4blgh" podStartSLOduration=37.735822252 podStartE2EDuration="45.107899564s" totalImagesPullingTime="7.372077312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:13.107818879 +0000 UTC m=+77.149866766" watchObservedRunningTime="2026-07-24 13:35:13.107899564 +0000 UTC m=+77.149947451"
Jul 24 13:35:13.165852 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.165790    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c946z" podStartSLOduration=35.479629324 podStartE2EDuration="42.165766628s" totalImagesPullingTime="6.686137304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:13.16147875 +0000 UTC m=+77.203526637" watchObservedRunningTime="2026-07-24 13:35:13.165766628 +0000 UTC m=+77.207814493"
Jul 24 13:35:13.212023 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:13.211959    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gdskt" podStartSLOduration=69.647706106 podStartE2EDuration="1m16.211917262s" totalImagesPullingTime="6.564211156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:13.196352747 +0000 UTC m=+77.238400633" watchObservedRunningTime="2026-07-24 13:35:13.211917262 +0000 UTC m=+77.253965152"
Jul 24 13:35:23.117594 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.117563    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c946z"
Jul 24 13:35:23.167390 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.167352    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x" event={"ID":"0e78c29d-a32a-4c0a-a2af-1d4dab69fbb9","Type":"ContainerStarted","Data":"c5e6f7c3e53423505112c0c31ec78caa5a5ddf42119ac3b51c1684a67892997f"}
Jul 24 13:35:23.167619 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.167418    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:23.169655 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.169619    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x"
Jul 24 13:35:23.170086 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.170051    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7" event={"ID":"964e98b6-6bfe-49cb-8e46-e64cc75ad9ae","Type":"ContainerStarted","Data":"0ca62794ace2e99ae9358021de5c57cfd8b4d6cde0b3d2f3cc07c68b1af9ad5d"}
Jul 24 13:35:23.179461 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.179412    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvln4" event={"ID":"de81ac99-198b-421c-88d4-998d005dcb61","Type":"ContainerStarted","Data":"03cd7e22304f0a7a9be1a17774a95e580ed2e01d0ce2cb3323c6d99fdc11e366"}
Jul 24 13:35:23.182004 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.181975    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8" event={"ID":"fd809ac8-485e-4736-ab74-cd3ffd2f3f41","Type":"ContainerStarted","Data":"2eb6757e37f554b325405537c2b96f4aeb5cf395c7debcebcf7e32ef9975bd1e"}
Jul 24 13:35:23.182144 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.182038    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8" event={"ID":"fd809ac8-485e-4736-ab74-cd3ffd2f3f41","Type":"ContainerStarted","Data":"b3be1cfa20b18c78521b873e4fba4e29758c991d21ede73bfc3b361e1f15209f"}
Jul 24 13:35:23.182144 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.182059    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8" event={"ID":"fd809ac8-485e-4736-ab74-cd3ffd2f3f41","Type":"ContainerStarted","Data":"0fb9c98f41fc5a2784fbac6c6cda6a8d893e6e7d1d458d10c943d3ea4b096049"}
Jul 24 13:35:23.183961 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.183939    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4" event={"ID":"d6c18a60-b9cf-4abc-9962-5e35020d8adc","Type":"ContainerStarted","Data":"a38ba59e536cc690446a9c8620b9cc8f378311494baf638cea5bdf89c3c8c55a"}
Jul 24 13:35:23.228910 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.228862    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-74xr7" podStartSLOduration=5.531065278 podStartE2EDuration="16.228844701s" totalImagesPullingTime="10.697779423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.227054565 +0000 UTC m=+87.269102451" watchObservedRunningTime="2026-07-24 13:35:23.228844701 +0000 UTC m=+87.270892566"
Jul 24 13:35:23.229365 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.229334    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f85dd99f-mhg8x" podStartSLOduration=5.51271946 podStartE2EDuration="16.229323116s" totalImagesPullingTime="10.716603656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.204974452 +0000 UTC m=+87.247022337" watchObservedRunningTime="2026-07-24 13:35:23.229323116 +0000 UTC m=+87.271370995"
Jul 24 13:35:23.306032 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.305975    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rvln4" podStartSLOduration=17.70430048 podStartE2EDuration="41.30595637s" totalImagesPullingTime="23.60165589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.270042025 +0000 UTC m=+87.312089902" watchObservedRunningTime="2026-07-24 13:35:23.30595637 +0000 UTC m=+87.348004255"
Jul 24 13:35:23.351145 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.351080    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77dfd5764d-bs5v4" podStartSLOduration=5.652866152 podStartE2EDuration="16.351058445s" totalImagesPullingTime="10.698192293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.305138025 +0000 UTC m=+87.347185910" watchObservedRunningTime="2026-07-24 13:35:23.351058445 +0000 UTC m=+87.393106331"
Jul 24 13:35:23.378666 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.378566    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-db887f84b-hrvn8" podStartSLOduration=5.298273716 podStartE2EDuration="16.378544306s" totalImagesPullingTime="11.08027059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.350049419 +0000 UTC m=+87.392097306" watchObservedRunningTime="2026-07-24 13:35:23.378544306 +0000 UTC m=+87.420592195"
Jul 24 13:35:23.433425 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.433371    2527 patch_prober.go:28] interesting pod/image-registry-678d47fbf-87pvp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 13:35:23.433624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:23.433483    2527 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 13:35:25.879183 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.878577    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-97whs"]
Jul 24 13:35:25.885534 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.885507    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:25.890267 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.890241    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mwmx5\""
Jul 24 13:35:25.890489 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.890474    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 13:35:25.890734 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.890716    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 13:35:25.891082 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.890951    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 13:35:25.891825 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:25.891414    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 13:35:26.012034 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.011306    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:26.056358 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056310    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-metrics-client-ca\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056398    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-root\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056428    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056500    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5x66\" (UniqueName: \"kubernetes.io/projected/862816b5-7915-40a1-b883-e1b84b70aae5-kube-api-access-t5x66\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056713 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056563    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-textfile\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056713 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056590    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-accelerators-collector-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056713 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056660    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-tls\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056713 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056709    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-sys\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.056895 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.056737    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-wtmp\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158219 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158139    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-sys\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158372 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158221    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-wtmp\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158372 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158278    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-metrics-client-ca\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158372 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158330    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-root\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158372 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158355    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158399    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5x66\" (UniqueName: \"kubernetes.io/projected/862816b5-7915-40a1-b883-e1b84b70aae5-kube-api-access-t5x66\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158479    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-textfile\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158506    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-accelerators-collector-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.158624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.158547    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-tls\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.159062 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.159034    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-wtmp\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.159157 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.159071    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-metrics-client-ca\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.159157 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.159105    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-sys\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.159938 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.159619    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/862816b5-7915-40a1-b883-e1b84b70aae5-root\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.159938 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.159875    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-textfile\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.160176 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.160151    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-accelerators-collector-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.162060 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.162017    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-tls\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.162426 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.162405    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/862816b5-7915-40a1-b883-e1b84b70aae5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.171750 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.171694    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5x66\" (UniqueName: \"kubernetes.io/projected/862816b5-7915-40a1-b883-e1b84b70aae5-kube-api-access-t5x66\") pod \"node-exporter-97whs\" (UID: \"862816b5-7915-40a1-b883-e1b84b70aae5\") " pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:26.200875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:26.200617    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97whs"
Jul 24 13:35:30.874881 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:30.874844    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:35:31.882455 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:35:31.882334    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862816b5_7915_40a1_b883_e1b84b70aae5.slice/crio-54eef661de62e20f14b252dff1956d3c29a1c7099a4d35631beeda0fac01c0f4 WatchSource:0}: Error finding container 54eef661de62e20f14b252dff1956d3c29a1c7099a4d35631beeda0fac01c0f4: Status 404 returned error can't find the container with id 54eef661de62e20f14b252dff1956d3c29a1c7099a4d35631beeda0fac01c0f4
Jul 24 13:35:32.228551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:32.228522    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97whs" event={"ID":"862816b5-7915-40a1-b883-e1b84b70aae5","Type":"ContainerStarted","Data":"54eef661de62e20f14b252dff1956d3c29a1c7099a4d35631beeda0fac01c0f4"}
Jul 24 13:35:33.235615 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:33.235567    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-t6vhk" event={"ID":"a2a0e067-907f-4aba-9145-5fc654769f71","Type":"ContainerStarted","Data":"21cd3a2a753109edb900ae6199219a76f8fdca73c721b7147b215861aaf47136"}
Jul 24 13:35:33.236103 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:33.235799    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:33.252997 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:33.252963    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-t6vhk"
Jul 24 13:35:33.255865 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:33.255784    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-t6vhk" podStartSLOduration=5.979491201 podStartE2EDuration="26.255766051s" totalImagesPullingTime="20.27627485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:33.253481188 +0000 UTC m=+97.295529074" watchObservedRunningTime="2026-07-24 13:35:33.255766051 +0000 UTC m=+97.297813938"
Jul 24 13:35:34.243211 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:34.243179    2527 generic.go:356] "Generic (PLEG): container finished" podID="862816b5-7915-40a1-b883-e1b84b70aae5" containerID="ec79d8c1baaedb4d8e9e61074c453b29cea601e9f433c6eba6d3fa726586514b" exitCode=0
Jul 24 13:35:34.243729 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:34.243262    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97whs" event={"ID":"862816b5-7915-40a1-b883-e1b84b70aae5","Type":"ContainerDied","Data":"ec79d8c1baaedb4d8e9e61074c453b29cea601e9f433c6eba6d3fa726586514b"}
Jul 24 13:35:35.258103 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:35.258062    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97whs" event={"ID":"862816b5-7915-40a1-b883-e1b84b70aae5","Type":"ContainerStarted","Data":"0d60734addf4d8322fed14367f7d116f57e56340e0c71644de897a1ff8878321"}
Jul 24 13:35:35.258598 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:35.258166    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97whs" event={"ID":"862816b5-7915-40a1-b883-e1b84b70aae5","Type":"ContainerStarted","Data":"ea7805e8f0b629dcee9639fc8f80f0ee0859bd96bc4db07332dc3264b2520999"}
Jul 24 13:35:35.282465 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:35.282383    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-97whs" podStartSLOduration=8.560883719 podStartE2EDuration="10.282362176s" totalImagesPullingTime="1.597968525s" totalInitContainerRuntime="123.509932ms" isStatefulPod=true podCreationTimestamp="2026-07-24 13:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:35.280689945 +0000 UTC m=+99.322737830" watchObservedRunningTime="2026-07-24 13:35:35.282362176 +0000 UTC m=+99.324410059"
Jul 24 13:35:36.008243 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:36.008205    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rj5bn"
Jul 24 13:35:52.330628 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:52.330588    2527 generic.go:356] "Generic (PLEG): container finished" podID="80506080-1458-4891-ba70-00afb1e62048" containerID="648317d76c868f72940800e181dfb32eff07e19c25bc3c16101be604588b12b5" exitCode=0
Jul 24 13:35:52.331116 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:52.330669    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd" event={"ID":"80506080-1458-4891-ba70-00afb1e62048","Type":"ContainerDied","Data":"648317d76c868f72940800e181dfb32eff07e19c25bc3c16101be604588b12b5"}
Jul 24 13:35:52.331116 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:52.331002    2527 scope.go:117] "RemoveContainer" containerID="648317d76c868f72940800e181dfb32eff07e19c25bc3c16101be604588b12b5"
Jul 24 13:35:53.336697 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:53.336667    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-s7xjd" event={"ID":"80506080-1458-4891-ba70-00afb1e62048","Type":"ContainerStarted","Data":"987c230992bf987a976baff938eed3f33f8c0dd3b153f34174153d03a32f9fbe"}
Jul 24 13:35:55.900286 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:55.900227    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry" containerID="cri-o://37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9" gracePeriod=30
Jul 24 13:35:56.005514 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.005478    2527 patch_prober.go:28] interesting pod/image-registry-678d47fbf-87pvp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body=
Jul 24 13:35:56.005665 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.005547    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused"
Jul 24 13:35:56.184633 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.184597    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:56.338690 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338654    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338702    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338729    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338767    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338803    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338832    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.338873 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338870    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4982t\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.339425 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.338947    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets\") pod \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\" (UID: \"6ed725e5-f396-4aaf-beed-dee1fc3b57b6\") "
Jul 24 13:35:56.339425 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.339368    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:35:56.339605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.339447    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:35:56.341888 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.341860    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:35:56.342014 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.341860    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:35:56.342077 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.342033    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:35:56.342191 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.342166    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t" (OuterVolumeSpecName: "kube-api-access-4982t") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "kube-api-access-4982t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:35:56.342258 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.342184    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:35:56.348628 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.348598    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6ed725e5-f396-4aaf-beed-dee1fc3b57b6" (UID: "6ed725e5-f396-4aaf-beed-dee1fc3b57b6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:35:56.352546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.352514    2527 generic.go:356] "Generic (PLEG): container finished" podID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerID="37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9" exitCode=0
Jul 24 13:35:56.352665 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.352606    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" event={"ID":"6ed725e5-f396-4aaf-beed-dee1fc3b57b6","Type":"ContainerDied","Data":"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"}
Jul 24 13:35:56.352728 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.352685    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d47fbf-87pvp" event={"ID":"6ed725e5-f396-4aaf-beed-dee1fc3b57b6","Type":"ContainerDied","Data":"7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765"}
Jul 24 13:35:56.352728 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.352722    2527 scope.go:117] "RemoveContainer" containerID="37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"
Jul 24 13:35:56.352937 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.352921    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d47fbf-87pvp"
Jul 24 13:35:56.363462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.363414    2527 scope.go:117] "RemoveContainer" containerID="37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"
Jul 24 13:35:56.363790 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:56.363766    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9\": container with ID starting with 37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9 not found: ID does not exist" containerID="37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"
Jul 24 13:35:56.363876 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.363803    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9"} err="failed to get container status \"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9\": rpc error: code = NotFound desc = could not find container \"37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9\": container with ID starting with 37d5526f4940f558593122c73480a13bfe4ca86fd5cba8d1ec45f5286f2c7fb9 not found: ID does not exist"
Jul 24 13:35:56.382508 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.382475    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:35:56.387255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.387231    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-678d47fbf-87pvp"]
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439490    2527 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-image-registry-private-configuration\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439523    2527 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-ca-trust-extracted\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439537    2527 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-certificates\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439551    2527 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-registry-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439563    2527 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-trusted-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439582 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439575    2527 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-bound-sa-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439962 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439587    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4982t\" (UniqueName: \"kubernetes.io/projected/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-kube-api-access-4982t\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.439962 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.439601    2527 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed725e5-f396-4aaf-beed-dee1fc3b57b6-installation-pull-secrets\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.513190 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:56.513149    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" path="/var/lib/kubelet/pods/6ed725e5-f396-4aaf-beed-dee1fc3b57b6/volumes"
Jul 24 13:35:56.745471 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:35:56.745353    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:35:57.358493 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:57.358394    2527 generic.go:356] "Generic (PLEG): container finished" podID="1587d831-aefd-4241-8f63-e0a31867f65d" containerID="67e5d6c597e908d22e717130c26601670c00ff059f08ee53526dfa19458340d5" exitCode=0
Jul 24 13:35:57.358493 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:57.358469    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7896g" event={"ID":"1587d831-aefd-4241-8f63-e0a31867f65d","Type":"ContainerDied","Data":"67e5d6c597e908d22e717130c26601670c00ff059f08ee53526dfa19458340d5"}
Jul 24 13:35:57.359050 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:57.358882    2527 scope.go:117] "RemoveContainer" containerID="67e5d6c597e908d22e717130c26601670c00ff059f08ee53526dfa19458340d5"
Jul 24 13:35:58.365482 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:35:58.365428    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7896g" event={"ID":"1587d831-aefd-4241-8f63-e0a31867f65d","Type":"ContainerStarted","Data":"f897df417b3f11f625397513eced9d7ca979e67bc311b94985f5d3a1cf6d660e"}
Jul 24 13:36:02.381729 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:36:02.381693    2527 generic.go:356] "Generic (PLEG): container finished" podID="65b2a17c-563e-4e60-868b-41f51530c308" containerID="0a27bf04dacc5653ab9e0b1392caac73d4a2445abb60483051c1199a092ebd1d" exitCode=0
Jul 24 13:36:02.382218 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:36:02.381774    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf" event={"ID":"65b2a17c-563e-4e60-868b-41f51530c308","Type":"ContainerDied","Data":"0a27bf04dacc5653ab9e0b1392caac73d4a2445abb60483051c1199a092ebd1d"}
Jul 24 13:36:02.382218 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:36:02.382184    2527 scope.go:117] "RemoveContainer" containerID="0a27bf04dacc5653ab9e0b1392caac73d4a2445abb60483051c1199a092ebd1d"
Jul 24 13:36:03.387701 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:36:03.387662    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf" event={"ID":"65b2a17c-563e-4e60-868b-41f51530c308","Type":"ContainerStarted","Data":"26c23cbecc0a56e9d9790bcc5303afa73cdc4039e2c59aba65a25532f2805a93"}
Jul 24 13:36:03.870225 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:03.870170    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:06.754358 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:06.754316    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:16.794152 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:16.794114    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:18.838094 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:18.838049    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:26.838126 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:26.838072    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:30.319185 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:30.319140    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:30.319637 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:30.319609    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:33.891208 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:33.891165    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:36.851284 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:36.851246    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:46.888864 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:46.888827    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:36:48.845167 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:36:48.845128    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice/crio-7334fd30b60710fdfe902b02a957302460b3934615b59a955e0555882d01f765\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed725e5_f396_4aaf_beed_dee1fc3b57b6.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 13:38:56.342603 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:38:56.342576    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:38:56.351795 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:38:56.351764    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:38:56.353276 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:38:56.353241    2527 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 13:40:03.323946 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.323855    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"]
Jul 24 13:40:03.324348 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.324210    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry"
Jul 24 13:40:03.324348 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.324224    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry"
Jul 24 13:40:03.324348 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.324283    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed725e5-f396-4aaf-beed-dee1fc3b57b6" containerName="registry"
Jul 24 13:40:03.326634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.326619    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.330320 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.330295    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 13:40:03.330480 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.330301    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 13:40:03.330548 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.330532    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 13:40:03.331010 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.330996    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fjml2\""
Jul 24 13:40:03.334631 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.334614    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 13:40:03.356720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.356685    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"]
Jul 24 13:40:03.412080 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.412041    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pv9\" (UniqueName: \"kubernetes.io/projected/699813cd-8368-4600-93e8-dc3a0dfef5f4-kube-api-access-88pv9\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.412266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.412111    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.412266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.412140    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.412266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.412155    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.412266 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.412174    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.512972 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.512935    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.513145 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.512983    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.513145 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.513105    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.513221 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.513152    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.513221 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.513210    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pv9\" (UniqueName: \"kubernetes.io/projected/699813cd-8368-4600-93e8-dc3a0dfef5f4-kube-api-access-88pv9\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.513557 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.513530    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.515491 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.515467    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699813cd-8368-4600-93e8-dc3a0dfef5f4-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.515745 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.515724    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.515803 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.515790    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/699813cd-8368-4600-93e8-dc3a0dfef5f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.528096 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.528064    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pv9\" (UniqueName: \"kubernetes.io/projected/699813cd-8368-4600-93e8-dc3a0dfef5f4-kube-api-access-88pv9\") pod \"opendatahub-operator-controller-manager-85fcbb566-52tsb\" (UID: \"699813cd-8368-4600-93e8-dc3a0dfef5f4\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.637231 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.637142    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:03.793952 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.793919    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"]
Jul 24 13:40:03.797636 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:40:03.797610    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699813cd_8368_4600_93e8_dc3a0dfef5f4.slice/crio-e631e1751d4db9fcb29958b4db812347f5bf77454322755fabb8bf009768df1e WatchSource:0}: Error finding container e631e1751d4db9fcb29958b4db812347f5bf77454322755fabb8bf009768df1e: Status 404 returned error can't find the container with id e631e1751d4db9fcb29958b4db812347f5bf77454322755fabb8bf009768df1e
Jul 24 13:40:03.799351 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:03.799330    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:40:04.316519 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:04.316486    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb" event={"ID":"699813cd-8368-4600-93e8-dc3a0dfef5f4","Type":"ContainerStarted","Data":"e631e1751d4db9fcb29958b4db812347f5bf77454322755fabb8bf009768df1e"}
Jul 24 13:40:07.333745 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:07.333709    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb" event={"ID":"699813cd-8368-4600-93e8-dc3a0dfef5f4","Type":"ContainerStarted","Data":"f81c797df0a0aea77afe654c859ad2ead58ce28762ae69ad3e50ceddb8dadf5d"}
Jul 24 13:40:08.339444 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:08.339406    2527 generic.go:356] "Generic (PLEG): container finished" podID="699813cd-8368-4600-93e8-dc3a0dfef5f4" containerID="f81c797df0a0aea77afe654c859ad2ead58ce28762ae69ad3e50ceddb8dadf5d" exitCode=0
Jul 24 13:40:08.339870 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:08.339492    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb" event={"ID":"699813cd-8368-4600-93e8-dc3a0dfef5f4","Type":"ContainerDied","Data":"f81c797df0a0aea77afe654c859ad2ead58ce28762ae69ad3e50ceddb8dadf5d"}
Jul 24 13:40:09.345685 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:09.345653    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb" event={"ID":"699813cd-8368-4600-93e8-dc3a0dfef5f4","Type":"ContainerStarted","Data":"a5cc1d4d6bd970514001545dbac086d6b132d82984f4f74e51602fdf056cbb8e"}
Jul 24 13:40:09.346078 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:09.345801    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:09.409659 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:09.409608    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb" podStartSLOduration=2.044446138 podStartE2EDuration="6.409588678s" totalImagesPullingTime="3.31555383s" totalInitContainerRuntime="1.04958871s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:09.408158418 +0000 UTC m=+373.450206314" watchObservedRunningTime="2026-07-24 13:40:09.409588678 +0000 UTC m=+373.451636563"
Jul 24 13:40:20.352722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:20.352694    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-52tsb"
Jul 24 13:40:23.573357 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.573325    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"]
Jul 24 13:40:23.581995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.581966    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.585186 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.585159    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zm78x\""
Jul 24 13:40:23.585409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.585383    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 13:40:23.585710 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.585223    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 13:40:23.591557 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.591480    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 13:40:23.591709 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.591511    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:40:23.592262 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.592234    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 13:40:23.599351 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.599326    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"]
Jul 24 13:40:23.683533 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.683487    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b73012c-00af-45d3-8b00-e17fbaa75141-manager-config\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.683718 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.683554    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-metrics-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.683718 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.683668    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpl8g\" (UniqueName: \"kubernetes.io/projected/9b73012c-00af-45d3-8b00-e17fbaa75141-kube-api-access-tpl8g\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.683808 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.683726    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.784535 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.784497    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpl8g\" (UniqueName: \"kubernetes.io/projected/9b73012c-00af-45d3-8b00-e17fbaa75141-kube-api-access-tpl8g\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.784720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.784545    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.784720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.784568    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b73012c-00af-45d3-8b00-e17fbaa75141-manager-config\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.784720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.784593    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-metrics-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.785333 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.785306    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b73012c-00af-45d3-8b00-e17fbaa75141-manager-config\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.787165 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.787135    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.787288 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.787223    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b73012c-00af-45d3-8b00-e17fbaa75141-metrics-cert\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.800674 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.800648    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpl8g\" (UniqueName: \"kubernetes.io/projected/9b73012c-00af-45d3-8b00-e17fbaa75141-kube-api-access-tpl8g\") pod \"lws-controller-manager-5d64c44479-rxtc2\" (UID: \"9b73012c-00af-45d3-8b00-e17fbaa75141\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:23.896720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:23.896625    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:24.060425 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:24.060396    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"]
Jul 24 13:40:24.063032 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:40:24.063004    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b73012c_00af_45d3_8b00_e17fbaa75141.slice/crio-6c85840d17ea717c0af2f3338d484b1ad9ef92ccc6944fef6730014074c99402 WatchSource:0}: Error finding container 6c85840d17ea717c0af2f3338d484b1ad9ef92ccc6944fef6730014074c99402: Status 404 returned error can't find the container with id 6c85840d17ea717c0af2f3338d484b1ad9ef92ccc6944fef6730014074c99402
Jul 24 13:40:24.411063 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:24.411025    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2" event={"ID":"9b73012c-00af-45d3-8b00-e17fbaa75141","Type":"ContainerStarted","Data":"6c85840d17ea717c0af2f3338d484b1ad9ef92ccc6944fef6730014074c99402"}
Jul 24 13:40:27.425965 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:27.425932    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2" event={"ID":"9b73012c-00af-45d3-8b00-e17fbaa75141","Type":"ContainerStarted","Data":"2e5ba1626531257eafc8d9516d3a5c68c2097eb253abcfa622f9be860a78e58b"}
Jul 24 13:40:27.426374 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:27.426041    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:40:27.443908 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:27.443854    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2" podStartSLOduration=2.046992837 podStartE2EDuration="4.44383054s" totalImagesPullingTime="2.396837703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:27.442715602 +0000 UTC m=+391.484763487" watchObservedRunningTime="2026-07-24 13:40:27.44383054 +0000 UTC m=+391.485878426"
Jul 24 13:40:38.433097 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:40:38.433066    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-rxtc2"
Jul 24 13:41:24.209802 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.209723    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:24.212981 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.212958    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:24.216090 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.216067    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 13:41:24.216553 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.216534    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sfmk8\""
Jul 24 13:41:24.216683 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.216584    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 13:41:24.232920 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.232886    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:24.295727 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.295690    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j6b\" (UniqueName: \"kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b\") pod \"kuadrant-operator-catalog-b5lql\" (UID: \"98ba467e-aba6-4a1b-b8e1-df0092d0ded7\") " pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:24.396735 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.396685    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j6b\" (UniqueName: \"kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b\") pod \"kuadrant-operator-catalog-b5lql\" (UID: \"98ba467e-aba6-4a1b-b8e1-df0092d0ded7\") " pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:24.406260 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.406236    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j6b\" (UniqueName: \"kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b\") pod \"kuadrant-operator-catalog-b5lql\" (UID: \"98ba467e-aba6-4a1b-b8e1-df0092d0ded7\") " pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:24.521699 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.521616    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:24.521887 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.521874    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:24.662781 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.662756    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:24.665169 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:41:24.665141    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ba467e_aba6_4a1b_b8e1_df0092d0ded7.slice/crio-4ae68dd6eb4ea77b5980c14ba2ab069605ef386043fa3a943f74a5bab2728040 WatchSource:0}: Error finding container 4ae68dd6eb4ea77b5980c14ba2ab069605ef386043fa3a943f74a5bab2728040: Status 404 returned error can't find the container with id 4ae68dd6eb4ea77b5980c14ba2ab069605ef386043fa3a943f74a5bab2728040
Jul 24 13:41:24.722444 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.722396    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9q2k"]
Jul 24 13:41:24.726645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.726628    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:24.742821 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.742792    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9q2k"]
Jul 24 13:41:24.799560 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.799465    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnts2\" (UniqueName: \"kubernetes.io/projected/7c0db250-bc4a-4986-9c7d-bfeee406ce43-kube-api-access-tnts2\") pod \"kuadrant-operator-catalog-k9q2k\" (UID: \"7c0db250-bc4a-4986-9c7d-bfeee406ce43\") " pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:24.899878 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.899829    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnts2\" (UniqueName: \"kubernetes.io/projected/7c0db250-bc4a-4986-9c7d-bfeee406ce43-kube-api-access-tnts2\") pod \"kuadrant-operator-catalog-k9q2k\" (UID: \"7c0db250-bc4a-4986-9c7d-bfeee406ce43\") " pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:24.909311 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:24.909280    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnts2\" (UniqueName: \"kubernetes.io/projected/7c0db250-bc4a-4986-9c7d-bfeee406ce43-kube-api-access-tnts2\") pod \"kuadrant-operator-catalog-k9q2k\" (UID: \"7c0db250-bc4a-4986-9c7d-bfeee406ce43\") " pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:25.042625 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:25.042589    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:25.197957 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:25.197924    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9q2k"]
Jul 24 13:41:25.201110 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:41:25.201080    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0db250_bc4a_4986_9c7d_bfeee406ce43.slice/crio-40702f070b6ee73e4041b1e2707edff62f80170ea7e79238eb98e1023f792f20 WatchSource:0}: Error finding container 40702f070b6ee73e4041b1e2707edff62f80170ea7e79238eb98e1023f792f20: Status 404 returned error can't find the container with id 40702f070b6ee73e4041b1e2707edff62f80170ea7e79238eb98e1023f792f20
Jul 24 13:41:25.671805 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:25.671768    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k" event={"ID":"7c0db250-bc4a-4986-9c7d-bfeee406ce43","Type":"ContainerStarted","Data":"40702f070b6ee73e4041b1e2707edff62f80170ea7e79238eb98e1023f792f20"}
Jul 24 13:41:25.673201 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:25.673155    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" event={"ID":"98ba467e-aba6-4a1b-b8e1-df0092d0ded7","Type":"ContainerStarted","Data":"4ae68dd6eb4ea77b5980c14ba2ab069605ef386043fa3a943f74a5bab2728040"}
Jul 24 13:41:27.683764 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.683730    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" event={"ID":"98ba467e-aba6-4a1b-b8e1-df0092d0ded7","Type":"ContainerStarted","Data":"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"}
Jul 24 13:41:27.684298 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.683777    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" podUID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" containerName="registry-server" containerID="cri-o://98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b" gracePeriod=2
Jul 24 13:41:27.685408 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.685379    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k" event={"ID":"7c0db250-bc4a-4986-9c7d-bfeee406ce43","Type":"ContainerStarted","Data":"ea48845f5c3475eb82c015fa708a95567863f3e86dd9160756b90f1fc582b7e2"}
Jul 24 13:41:27.710950 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.710899    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" podStartSLOduration=1.187032586 podStartE2EDuration="3.710882457s" totalImagesPullingTime="2.523849871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:27.710055325 +0000 UTC m=+451.752103213" watchObservedRunningTime="2026-07-24 13:41:27.710882457 +0000 UTC m=+451.752930341"
Jul 24 13:41:27.712371 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:41:27.712347    2527 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ba467e_aba6_4a1b_b8e1_df0092d0ded7.slice/crio-98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b.scope\": RecentStats: unable to find data in memory cache]"
Jul 24 13:41:27.729220 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.729171    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k" podStartSLOduration=1.715849688 podStartE2EDuration="3.729153117s" totalImagesPullingTime="2.013303429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:27.728140864 +0000 UTC m=+451.770188751" watchObservedRunningTime="2026-07-24 13:41:27.729153117 +0000 UTC m=+451.771201046"
Jul 24 13:41:27.939958 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:27.939891    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:28.031260 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.031210    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6j6b\" (UniqueName: \"kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b\") pod \"98ba467e-aba6-4a1b-b8e1-df0092d0ded7\" (UID: \"98ba467e-aba6-4a1b-b8e1-df0092d0ded7\") "
Jul 24 13:41:28.033575 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.033544    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b" (OuterVolumeSpecName: "kube-api-access-r6j6b") pod "98ba467e-aba6-4a1b-b8e1-df0092d0ded7" (UID: "98ba467e-aba6-4a1b-b8e1-df0092d0ded7"). InnerVolumeSpecName "kube-api-access-r6j6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:41:28.132276 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.132229    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6j6b\" (UniqueName: \"kubernetes.io/projected/98ba467e-aba6-4a1b-b8e1-df0092d0ded7-kube-api-access-r6j6b\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:41:28.696561 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.696523    2527 generic.go:356] "Generic (PLEG): container finished" podID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" containerID="98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b" exitCode=0
Jul 24 13:41:28.696976 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.696587    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b5lql"
Jul 24 13:41:28.696976 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.696611    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" event={"ID":"98ba467e-aba6-4a1b-b8e1-df0092d0ded7","Type":"ContainerDied","Data":"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"}
Jul 24 13:41:28.696976 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.696688    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b5lql" event={"ID":"98ba467e-aba6-4a1b-b8e1-df0092d0ded7","Type":"ContainerDied","Data":"4ae68dd6eb4ea77b5980c14ba2ab069605ef386043fa3a943f74a5bab2728040"}
Jul 24 13:41:28.696976 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.696712    2527 scope.go:117] "RemoveContainer" containerID="98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"
Jul 24 13:41:28.705824 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.705802    2527 scope.go:117] "RemoveContainer" containerID="98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"
Jul 24 13:41:28.706089 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:41:28.706068    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b\": container with ID starting with 98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b not found: ID does not exist" containerID="98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"
Jul 24 13:41:28.706152 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.706098    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b"} err="failed to get container status \"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b\": rpc error: code = NotFound desc = could not find container \"98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b\": container with ID starting with 98fe1855c8f16c0dfa3cefdf9c82d3e1a79b8b4e8a3fea9d0809ee5533e8710b not found: ID does not exist"
Jul 24 13:41:28.714271 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.714244    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:28.718787 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:28.718762    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b5lql"]
Jul 24 13:41:30.512826 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:30.512794    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" path="/var/lib/kubelet/pods/98ba467e-aba6-4a1b-b8e1-df0092d0ded7/volumes"
Jul 24 13:41:35.042948 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:35.042909    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:35.043350 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:35.043066    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:35.065157 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:35.065126    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:35.748960 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:35.748920    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-k9q2k"
Jul 24 13:41:57.347816 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.347768    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"]
Jul 24 13:41:57.348205 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.348097    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" containerName="registry-server"
Jul 24 13:41:57.348205 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.348108    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" containerName="registry-server"
Jul 24 13:41:57.348205 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.348169    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ba467e-aba6-4a1b-b8e1-df0092d0ded7" containerName="registry-server"
Jul 24 13:41:57.352289 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.352268    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:41:57.355568 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.355521    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l9t8j\""
Jul 24 13:41:57.370743 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.370702    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"]
Jul 24 13:41:57.384420 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.384378    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmtb\" (UniqueName: \"kubernetes.io/projected/b01cd528-b42d-4d11-a626-838163db18e0-kube-api-access-xhmtb\") pod \"authorino-operator-75cc8d5cb7-7knf2\" (UID: \"b01cd528-b42d-4d11-a626-838163db18e0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:41:57.485234 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.485190    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmtb\" (UniqueName: \"kubernetes.io/projected/b01cd528-b42d-4d11-a626-838163db18e0-kube-api-access-xhmtb\") pod \"authorino-operator-75cc8d5cb7-7knf2\" (UID: \"b01cd528-b42d-4d11-a626-838163db18e0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:41:57.509619 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.509580    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmtb\" (UniqueName: \"kubernetes.io/projected/b01cd528-b42d-4d11-a626-838163db18e0-kube-api-access-xhmtb\") pod \"authorino-operator-75cc8d5cb7-7knf2\" (UID: \"b01cd528-b42d-4d11-a626-838163db18e0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:41:57.662212 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.662116    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:41:57.816588 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:57.816558    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"]
Jul 24 13:41:58.825445 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:41:58.825405    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2" event={"ID":"b01cd528-b42d-4d11-a626-838163db18e0","Type":"ContainerStarted","Data":"6a0e1dbabf84c948a402933c5303c51772a9cb22437074e072e83b333f91ec4c"}
Jul 24 13:42:00.836302 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:00.836268    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2" event={"ID":"b01cd528-b42d-4d11-a626-838163db18e0","Type":"ContainerStarted","Data":"37c0689ae13850c1e3e08969b5f3486efc8c9ed7ffd0648ee997f31d38386375"}
Jul 24 13:42:00.836783 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:00.836418    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:42:00.857673 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:00.857618    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2" podStartSLOduration=1.45747462 podStartE2EDuration="3.857587588s" totalImagesPullingTime="2.400112968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:00.857068233 +0000 UTC m=+484.899116118" watchObservedRunningTime="2026-07-24 13:42:00.857587588 +0000 UTC m=+484.899635473"
Jul 24 13:42:07.522080 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.522041    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"]
Jul 24 13:42:07.528600 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.528575    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:07.530547 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.530512    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 24 13:42:07.530655 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.530544    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vvzd6\""
Jul 24 13:42:07.539107 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.539081    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"]
Jul 24 13:42:07.569662 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.569624    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbv8z\" (UniqueName: \"kubernetes.io/projected/94e55bc5-2306-49b9-bd02-b1e3d1504143-kube-api-access-rbv8z\") pod \"dns-operator-controller-manager-864dd78c4c-8jjmp\" (UID: \"94e55bc5-2306-49b9-bd02-b1e3d1504143\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:07.670678 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.670639    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbv8z\" (UniqueName: \"kubernetes.io/projected/94e55bc5-2306-49b9-bd02-b1e3d1504143-kube-api-access-rbv8z\") pod \"dns-operator-controller-manager-864dd78c4c-8jjmp\" (UID: \"94e55bc5-2306-49b9-bd02-b1e3d1504143\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:07.680621 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.680582    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbv8z\" (UniqueName: \"kubernetes.io/projected/94e55bc5-2306-49b9-bd02-b1e3d1504143-kube-api-access-rbv8z\") pod \"dns-operator-controller-manager-864dd78c4c-8jjmp\" (UID: \"94e55bc5-2306-49b9-bd02-b1e3d1504143\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:07.839412 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.839374    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:07.984170 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:07.984133    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"]
Jul 24 13:42:07.987241 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:42:07.987211    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e55bc5_2306_49b9_bd02_b1e3d1504143.slice/crio-2ed5ba975e3ed93a382b27dcefc7005b3e838115814501829765c148393fb6f0 WatchSource:0}: Error finding container 2ed5ba975e3ed93a382b27dcefc7005b3e838115814501829765c148393fb6f0: Status 404 returned error can't find the container with id 2ed5ba975e3ed93a382b27dcefc7005b3e838115814501829765c148393fb6f0
Jul 24 13:42:08.872738 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:08.872701    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp" event={"ID":"94e55bc5-2306-49b9-bd02-b1e3d1504143","Type":"ContainerStarted","Data":"2ed5ba975e3ed93a382b27dcefc7005b3e838115814501829765c148393fb6f0"}
Jul 24 13:42:10.883726 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:10.883693    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp" event={"ID":"94e55bc5-2306-49b9-bd02-b1e3d1504143","Type":"ContainerStarted","Data":"8959295e7ba86343751e6a4380a41b4482ccd89a51f6a62817beaa1818b625e2"}
Jul 24 13:42:10.884114 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:10.883763    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:10.906415 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:10.906357    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp" podStartSLOduration=1.7586948900000001 podStartE2EDuration="3.906341569s" totalImagesPullingTime="2.147646679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:10.90557201 +0000 UTC m=+494.947619902" watchObservedRunningTime="2026-07-24 13:42:10.906341569 +0000 UTC m=+494.948389454"
Jul 24 13:42:11.843616 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:11.843582    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7knf2"
Jul 24 13:42:21.891799 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:21.891758    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8jjmp"
Jul 24 13:42:53.089018 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.088938    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:42:53.092245 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.092226    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.094117 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.094096    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wbnz6\""
Jul 24 13:42:53.094252 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.094097    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 13:42:53.101054 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.101025    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:42:53.166349 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.166315    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.166559 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.166364    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szfp\" (UniqueName: \"kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.183478 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.183424    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:42:53.267274 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.267233    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.267497 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.267301    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szfp\" (UniqueName: \"kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.267857 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.267836    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.275780 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.275758    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szfp\" (UniqueName: \"kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp\") pod \"limitador-limitador-64c94fd64f-chcdb\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.403826 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.403730    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:53.544890 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:53.544864    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:42:53.547814 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:42:53.547783    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23892ffd_8d2a_47e6_8c3e_f5fdc428dc61.slice/crio-2e24e9f810bf06e0d76a14cbd8e11ce3ee39ca7e2de1f4d555dc01207786050c WatchSource:0}: Error finding container 2e24e9f810bf06e0d76a14cbd8e11ce3ee39ca7e2de1f4d555dc01207786050c: Status 404 returned error can't find the container with id 2e24e9f810bf06e0d76a14cbd8e11ce3ee39ca7e2de1f4d555dc01207786050c
Jul 24 13:42:54.075419 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.075382    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" event={"ID":"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61","Type":"ContainerStarted","Data":"2e24e9f810bf06e0d76a14cbd8e11ce3ee39ca7e2de1f4d555dc01207786050c"}
Jul 24 13:42:54.081655 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.081629    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:42:54.085903 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.085881    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:42:54.087678 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.087654    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-52l8r\""
Jul 24 13:42:54.091541 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.091514    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:42:54.173799 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.173768    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr7b\" (UniqueName: \"kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b\") pod \"authorino-75d556bd77-6j5vt\" (UID: \"cfa35687-1771-42bc-8ddb-a105026e9001\") " pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:42:54.275191 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.275155    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr7b\" (UniqueName: \"kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b\") pod \"authorino-75d556bd77-6j5vt\" (UID: \"cfa35687-1771-42bc-8ddb-a105026e9001\") " pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:42:54.283958 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.283925    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr7b\" (UniqueName: \"kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b\") pod \"authorino-75d556bd77-6j5vt\" (UID: \"cfa35687-1771-42bc-8ddb-a105026e9001\") " pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:42:54.396985 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.396868    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:42:54.586166 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:54.586105    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:42:55.085424 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:55.085378    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6j5vt" event={"ID":"cfa35687-1771-42bc-8ddb-a105026e9001","Type":"ContainerStarted","Data":"c7f773d0a458fff6487b45682ea1dccf9f8739d5c6c1c630045e663c5f1b74f7"}
Jul 24 13:42:59.106421 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:59.106384    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" event={"ID":"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61","Type":"ContainerStarted","Data":"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"}
Jul 24 13:42:59.106896 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:59.106485    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:42:59.107910 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:59.107886    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6j5vt" event={"ID":"cfa35687-1771-42bc-8ddb-a105026e9001","Type":"ContainerStarted","Data":"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"}
Jul 24 13:42:59.124241 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:59.124187    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" podStartSLOduration=1.362730332 podStartE2EDuration="6.124168456s" totalImagesPullingTime="4.761438124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:59.123477846 +0000 UTC m=+543.165525826" watchObservedRunningTime="2026-07-24 13:42:59.124168456 +0000 UTC m=+543.166216340"
Jul 24 13:42:59.138652 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:42:59.138606    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-6j5vt" podStartSLOduration=1.4120125350000001 podStartE2EDuration="5.138588851s" totalImagesPullingTime="3.726576316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:59.137792265 +0000 UTC m=+543.179840149" watchObservedRunningTime="2026-07-24 13:42:59.138588851 +0000 UTC m=+543.180636737"
Jul 24 13:43:08.028799 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.028759    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:43:08.029315 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.028953    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" podUID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" containerName="limitador" containerID="cri-o://0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b" gracePeriod=30
Jul 24 13:43:08.029800 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.029779    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:43:08.584026 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.583990    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:43:08.702068 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.701972    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4szfp\" (UniqueName: \"kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp\") pod \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") "
Jul 24 13:43:08.702068 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.702028    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file\") pod \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\" (UID: \"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61\") "
Jul 24 13:43:08.702390 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.702366    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file" (OuterVolumeSpecName: "config-file") pod "23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" (UID: "23892ffd-8d2a-47e6-8c3e-f5fdc428dc61"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:43:08.704354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.704322    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp" (OuterVolumeSpecName: "kube-api-access-4szfp") pod "23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" (UID: "23892ffd-8d2a-47e6-8c3e-f5fdc428dc61"). InnerVolumeSpecName "kube-api-access-4szfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:08.803413 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.803368    2527 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-config-file\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:08.803413 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:08.803404    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szfp\" (UniqueName: \"kubernetes.io/projected/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61-kube-api-access-4szfp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:09.157668 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.157633    2527 generic.go:356] "Generic (PLEG): container finished" podID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" containerID="0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b" exitCode=0
Jul 24 13:43:09.158268 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.157675    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb"
Jul 24 13:43:09.158268 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.157723    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" event={"ID":"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61","Type":"ContainerDied","Data":"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"}
Jul 24 13:43:09.158268 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.157786    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-chcdb" event={"ID":"23892ffd-8d2a-47e6-8c3e-f5fdc428dc61","Type":"ContainerDied","Data":"2e24e9f810bf06e0d76a14cbd8e11ce3ee39ca7e2de1f4d555dc01207786050c"}
Jul 24 13:43:09.158268 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.157822    2527 scope.go:117] "RemoveContainer" containerID="0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"
Jul 24 13:43:09.166682 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.166662    2527 scope.go:117] "RemoveContainer" containerID="0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"
Jul 24 13:43:09.166937 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:43:09.166918    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b\": container with ID starting with 0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b not found: ID does not exist" containerID="0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"
Jul 24 13:43:09.166975 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.166946    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b"} err="failed to get container status \"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b\": rpc error: code = NotFound desc = could not find container \"0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b\": container with ID starting with 0b564211bcc4e2ffbde4275d34c1ab220640b1dda2c7b1b3d1d87ada3139059b not found: ID does not exist"
Jul 24 13:43:09.177550 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.177521    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:43:09.183270 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.183245    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-chcdb"]
Jul 24 13:43:09.672383 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.672348    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"]
Jul 24 13:43:09.672734 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.672717    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" containerName="limitador"
Jul 24 13:43:09.672734 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.672733    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" containerName="limitador"
Jul 24 13:43:09.672875 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.672811    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" containerName="limitador"
Jul 24 13:43:09.677247 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.677224    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.679318 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.679294    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 13:43:09.679457 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.679335    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:43:09.679457 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.679336    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5f5rc\""
Jul 24 13:43:09.679610 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.679597    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:43:09.683655 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.683631    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"]
Jul 24 13:43:09.712795 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.712749    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5df22bd4-67a2-4f30-b59d-742c70566fa3-data\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.712795 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.712790    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxl7\" (UniqueName: \"kubernetes.io/projected/5df22bd4-67a2-4f30-b59d-742c70566fa3-kube-api-access-5lxl7\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.814236 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.814196    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5df22bd4-67a2-4f30-b59d-742c70566fa3-data\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.814236 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.814235    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxl7\" (UniqueName: \"kubernetes.io/projected/5df22bd4-67a2-4f30-b59d-742c70566fa3-kube-api-access-5lxl7\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.814680 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.814660    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5df22bd4-67a2-4f30-b59d-742c70566fa3-data\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.830958 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.830924    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxl7\" (UniqueName: \"kubernetes.io/projected/5df22bd4-67a2-4f30-b59d-742c70566fa3-kube-api-access-5lxl7\") pod \"postgres-69fb4594cf-r7gbc\" (UID: \"5df22bd4-67a2-4f30-b59d-742c70566fa3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:09.988515 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:09.988400    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:10.126381 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:10.126346    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"]
Jul 24 13:43:10.129168 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:43:10.129139    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df22bd4_67a2_4f30_b59d_742c70566fa3.slice/crio-1b8b131d9ddacb4ca2a26889348598283031dbe72005862e82a6461f915dde96 WatchSource:0}: Error finding container 1b8b131d9ddacb4ca2a26889348598283031dbe72005862e82a6461f915dde96: Status 404 returned error can't find the container with id 1b8b131d9ddacb4ca2a26889348598283031dbe72005862e82a6461f915dde96
Jul 24 13:43:10.163848 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:10.163815    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc" event={"ID":"5df22bd4-67a2-4f30-b59d-742c70566fa3","Type":"ContainerStarted","Data":"1b8b131d9ddacb4ca2a26889348598283031dbe72005862e82a6461f915dde96"}
Jul 24 13:43:10.513503 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:10.513472    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23892ffd-8d2a-47e6-8c3e-f5fdc428dc61" path="/var/lib/kubelet/pods/23892ffd-8d2a-47e6-8c3e-f5fdc428dc61/volumes"
Jul 24 13:43:15.193670 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:15.193630    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc" event={"ID":"5df22bd4-67a2-4f30-b59d-742c70566fa3","Type":"ContainerStarted","Data":"20c582d797751ac7717247c34ce27fe01ec202d0f3e3ba17db74bd514842e6bd"}
Jul 24 13:43:15.194037 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:15.193736    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:15.210775 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:15.210723    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc" podStartSLOduration=1.3189940820000001 podStartE2EDuration="6.21070513s" totalImagesPullingTime="4.891711048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:15.209188776 +0000 UTC m=+559.251236660" watchObservedRunningTime="2026-07-24 13:43:15.21070513 +0000 UTC m=+559.252753015"
Jul 24 13:43:21.231024 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:21.230985    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r7gbc"
Jul 24 13:43:22.097338 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.097299    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:22.103544 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.103523    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:22.109941 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.109912    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:22.228801 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.228769    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvw46\" (UniqueName: \"kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46\") pod \"authorino-75bf8c6b8f-zgj95\" (UID: \"f05e7ccc-d052-406f-b759-d666086a1cfd\") " pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:22.330047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.330011    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvw46\" (UniqueName: \"kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46\") pod \"authorino-75bf8c6b8f-zgj95\" (UID: \"f05e7ccc-d052-406f-b759-d666086a1cfd\") " pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:22.338317 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.338276    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvw46\" (UniqueName: \"kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46\") pod \"authorino-75bf8c6b8f-zgj95\" (UID: \"f05e7ccc-d052-406f-b759-d666086a1cfd\") " pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:22.414060 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.413968    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:22.465552 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.465519    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:22.591324 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:22.591288    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:22.594208 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:43:22.594179    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf05e7ccc_d052_406f_b759_d666086a1cfd.slice/crio-110c6de889eca698815649c113083158083ef1664705204be6afefb0ce50a353 WatchSource:0}: Error finding container 110c6de889eca698815649c113083158083ef1664705204be6afefb0ce50a353: Status 404 returned error can't find the container with id 110c6de889eca698815649c113083158083ef1664705204be6afefb0ce50a353
Jul 24 13:43:23.233564 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.233527    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" event={"ID":"f05e7ccc-d052-406f-b759-d666086a1cfd","Type":"ContainerStarted","Data":"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"}
Jul 24 13:43:23.233737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.233576    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" event={"ID":"f05e7ccc-d052-406f-b759-d666086a1cfd","Type":"ContainerStarted","Data":"110c6de889eca698815649c113083158083ef1664705204be6afefb0ce50a353"}
Jul 24 13:43:23.233737 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.233611    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" podUID="f05e7ccc-d052-406f-b759-d666086a1cfd" containerName="authorino" containerID="cri-o://a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd" gracePeriod=30
Jul 24 13:43:23.248847 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.248798    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" podStartSLOduration=0.689251589 podStartE2EDuration="1.24878088s" totalImagesPullingTime="559.529291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:23.247129935 +0000 UTC m=+567.289177801" watchObservedRunningTime="2026-07-24 13:43:23.24878088 +0000 UTC m=+567.290828764"
Jul 24 13:43:23.488562 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.488535    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:23.640830 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.640797    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvw46\" (UniqueName: \"kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46\") pod \"f05e7ccc-d052-406f-b759-d666086a1cfd\" (UID: \"f05e7ccc-d052-406f-b759-d666086a1cfd\") "
Jul 24 13:43:23.643164 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.643138    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46" (OuterVolumeSpecName: "kube-api-access-dvw46") pod "f05e7ccc-d052-406f-b759-d666086a1cfd" (UID: "f05e7ccc-d052-406f-b759-d666086a1cfd"). InnerVolumeSpecName "kube-api-access-dvw46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:23.656680 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.656647    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:43:23.656890 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.656852    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-6j5vt" podUID="cfa35687-1771-42bc-8ddb-a105026e9001" containerName="authorino" containerID="cri-o://4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844" gracePeriod=30
Jul 24 13:43:23.742183 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.742098    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvw46\" (UniqueName: \"kubernetes.io/projected/f05e7ccc-d052-406f-b759-d666086a1cfd-kube-api-access-dvw46\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:23.898646 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:23.898618    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:43:24.045486 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.045379    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptr7b\" (UniqueName: \"kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b\") pod \"cfa35687-1771-42bc-8ddb-a105026e9001\" (UID: \"cfa35687-1771-42bc-8ddb-a105026e9001\") "
Jul 24 13:43:24.047639 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.047612    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b" (OuterVolumeSpecName: "kube-api-access-ptr7b") pod "cfa35687-1771-42bc-8ddb-a105026e9001" (UID: "cfa35687-1771-42bc-8ddb-a105026e9001"). InnerVolumeSpecName "kube-api-access-ptr7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:24.147001 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.146962    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr7b\" (UniqueName: \"kubernetes.io/projected/cfa35687-1771-42bc-8ddb-a105026e9001-kube-api-access-ptr7b\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:24.238953 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.238923    2527 generic.go:356] "Generic (PLEG): container finished" podID="cfa35687-1771-42bc-8ddb-a105026e9001" containerID="4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844" exitCode=0
Jul 24 13:43:24.239154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.238972    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6j5vt"
Jul 24 13:43:24.239154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.239017    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6j5vt" event={"ID":"cfa35687-1771-42bc-8ddb-a105026e9001","Type":"ContainerDied","Data":"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"}
Jul 24 13:43:24.239154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.239061    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6j5vt" event={"ID":"cfa35687-1771-42bc-8ddb-a105026e9001","Type":"ContainerDied","Data":"c7f773d0a458fff6487b45682ea1dccf9f8739d5c6c1c630045e663c5f1b74f7"}
Jul 24 13:43:24.239154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.239080    2527 scope.go:117] "RemoveContainer" containerID="4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"
Jul 24 13:43:24.240578 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.240540    2527 generic.go:356] "Generic (PLEG): container finished" podID="f05e7ccc-d052-406f-b759-d666086a1cfd" containerID="a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd" exitCode=2
Jul 24 13:43:24.240715 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.240591    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" event={"ID":"f05e7ccc-d052-406f-b759-d666086a1cfd","Type":"ContainerDied","Data":"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"}
Jul 24 13:43:24.240715 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.240623    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95" event={"ID":"f05e7ccc-d052-406f-b759-d666086a1cfd","Type":"ContainerDied","Data":"110c6de889eca698815649c113083158083ef1664705204be6afefb0ce50a353"}
Jul 24 13:43:24.240715 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.240596    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bf8c6b8f-zgj95"
Jul 24 13:43:24.249289 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.249267    2527 scope.go:117] "RemoveContainer" containerID="4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"
Jul 24 13:43:24.249650 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:43:24.249630    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844\": container with ID starting with 4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844 not found: ID does not exist" containerID="4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"
Jul 24 13:43:24.249725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.249659    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844"} err="failed to get container status \"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844\": rpc error: code = NotFound desc = could not find container \"4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844\": container with ID starting with 4725bc391f5c2724dbd751a1a3a39634cc71da799de51727e8801bcd37c01844 not found: ID does not exist"
Jul 24 13:43:24.249725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.249680    2527 scope.go:117] "RemoveContainer" containerID="a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"
Jul 24 13:43:24.258315 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.258298    2527 scope.go:117] "RemoveContainer" containerID="a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"
Jul 24 13:43:24.258630 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:43:24.258609    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd\": container with ID starting with a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd not found: ID does not exist" containerID="a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"
Jul 24 13:43:24.258694 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.258637    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd"} err="failed to get container status \"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd\": rpc error: code = NotFound desc = could not find container \"a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd\": container with ID starting with a6ba8a1d9d3cc1d453141ddd7fe5279e8701f81cfdc11a2f2f310df64218facd not found: ID does not exist"
Jul 24 13:43:24.263128 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.263105    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:43:24.269059 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.269025    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6j5vt"]
Jul 24 13:43:24.278550 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.278512    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:24.282783 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.282758    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75bf8c6b8f-zgj95"]
Jul 24 13:43:24.513840 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.513813    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa35687-1771-42bc-8ddb-a105026e9001" path="/var/lib/kubelet/pods/cfa35687-1771-42bc-8ddb-a105026e9001/volumes"
Jul 24 13:43:24.514240 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:24.514227    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f05e7ccc-d052-406f-b759-d666086a1cfd" path="/var/lib/kubelet/pods/f05e7ccc-d052-406f-b759-d666086a1cfd/volumes"
Jul 24 13:43:28.380313 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380279    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380645    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f05e7ccc-d052-406f-b759-d666086a1cfd" containerName="authorino"
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380657    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="f05e7ccc-d052-406f-b759-d666086a1cfd" containerName="authorino"
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380679    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa35687-1771-42bc-8ddb-a105026e9001" containerName="authorino"
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380684    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa35687-1771-42bc-8ddb-a105026e9001" containerName="authorino"
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380735    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="f05e7ccc-d052-406f-b759-d666086a1cfd" containerName="authorino"
Jul 24 13:43:28.380806 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.380746    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa35687-1771-42bc-8ddb-a105026e9001" containerName="authorino"
Jul 24 13:43:28.585634 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.585595    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.587771 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.587740    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 13:43:28.588183 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.588160    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 13:43:28.588312 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.588185    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mnksf\""
Jul 24 13:43:28.593067 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.593037    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:28.593225 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.593101    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:43:28.618795 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.618756    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:43:28.618991 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.618896    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.685411 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.685328    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.685411 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.685377    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.685411 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.685397    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tm7\" (UniqueName: \"kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.685663 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.685512    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdnx\" (UniqueName: \"kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.785941 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.785899    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.786141 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.785960    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.786141 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.785986    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tm7\" (UniqueName: \"kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.786141 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.786032    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdnx\" (UniqueName: \"kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.788723 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.788693    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.788836 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.788705    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.794088 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.794046    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tm7\" (UniqueName: \"kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7\") pod \"maas-controller-55fd855cd4-jwf8v\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") " pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.794246 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.794224    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdnx\" (UniqueName: \"kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx\") pod \"maas-controller-766d9c797c-fn4mg\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") " pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:28.897187 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.897144    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:28.929546 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:28.929521    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:29.097604 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:29.097502    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:43:29.100418 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:43:29.100390    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f7eadc_2adb_4b6b_a29c_c2d16f9332b0.slice/crio-532f81fbc2c00e226558c7e663b8c95d5e950c892d842efc98d57ed3c670ed78 WatchSource:0}: Error finding container 532f81fbc2c00e226558c7e663b8c95d5e950c892d842efc98d57ed3c670ed78: Status 404 returned error can't find the container with id 532f81fbc2c00e226558c7e663b8c95d5e950c892d842efc98d57ed3c670ed78
Jul 24 13:43:29.272771 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:29.272665    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-fn4mg" event={"ID":"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0","Type":"ContainerStarted","Data":"532f81fbc2c00e226558c7e663b8c95d5e950c892d842efc98d57ed3c670ed78"}
Jul 24 13:43:29.275547 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:29.275515    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:29.279396 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:43:29.279364    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd722b25f_2e16_46c8_90f9_f3e5e3455e58.slice/crio-28a069c2fe4ace28ff37b6820625d14afe9c52b290215187f291cacb6641a05e WatchSource:0}: Error finding container 28a069c2fe4ace28ff37b6820625d14afe9c52b290215187f291cacb6641a05e: Status 404 returned error can't find the container with id 28a069c2fe4ace28ff37b6820625d14afe9c52b290215187f291cacb6641a05e
Jul 24 13:43:30.293181 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:30.293140    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" event={"ID":"d722b25f-2e16-46c8-90f9-f3e5e3455e58","Type":"ContainerStarted","Data":"28a069c2fe4ace28ff37b6820625d14afe9c52b290215187f291cacb6641a05e"}
Jul 24 13:43:32.309047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.308943    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-fn4mg" event={"ID":"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0","Type":"ContainerStarted","Data":"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"}
Jul 24 13:43:32.309047 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.309012    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:32.310480 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.310452    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" event={"ID":"d722b25f-2e16-46c8-90f9-f3e5e3455e58","Type":"ContainerStarted","Data":"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"}
Jul 24 13:43:32.310631 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.310498    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:32.327423 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.327359    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766d9c797c-fn4mg" podStartSLOduration=1.586280741 podStartE2EDuration="4.327340469s" totalImagesPullingTime="2.741059728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:32.327011201 +0000 UTC m=+576.369059099" watchObservedRunningTime="2026-07-24 13:43:32.327340469 +0000 UTC m=+576.369388358"
Jul 24 13:43:32.349749 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:32.349692    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" podStartSLOduration=1.798070809 podStartE2EDuration="4.349673154s" totalImagesPullingTime="2.551602345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:32.348987052 +0000 UTC m=+576.391034941" watchObservedRunningTime="2026-07-24 13:43:32.349673154 +0000 UTC m=+576.391721038"
Jul 24 13:43:43.318252 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.318218    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:43.318676 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.318292    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:43:43.400109 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.400066    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:43.400323 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.400300    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" podUID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" containerName="manager" containerID="cri-o://444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9" gracePeriod=30
Jul 24 13:43:43.655445 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.655403    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:43.816441 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.816404    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert\") pod \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") "
Jul 24 13:43:43.816633 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.816488    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4tm7\" (UniqueName: \"kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7\") pod \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\" (UID: \"d722b25f-2e16-46c8-90f9-f3e5e3455e58\") "
Jul 24 13:43:43.818900 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.818867    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d722b25f-2e16-46c8-90f9-f3e5e3455e58" (UID: "d722b25f-2e16-46c8-90f9-f3e5e3455e58"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:43:43.818900 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.818871    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7" (OuterVolumeSpecName: "kube-api-access-p4tm7") pod "d722b25f-2e16-46c8-90f9-f3e5e3455e58" (UID: "d722b25f-2e16-46c8-90f9-f3e5e3455e58"). InnerVolumeSpecName "kube-api-access-p4tm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:43.918070 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.918032    2527 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d722b25f-2e16-46c8-90f9-f3e5e3455e58-webhook-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:43.918070 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:43.918065    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tm7\" (UniqueName: \"kubernetes.io/projected/d722b25f-2e16-46c8-90f9-f3e5e3455e58-kube-api-access-p4tm7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:43:44.368561 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.368523    2527 generic.go:356] "Generic (PLEG): container finished" podID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" containerID="444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9" exitCode=0
Jul 24 13:43:44.369044 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.368593    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55fd855cd4-jwf8v"
Jul 24 13:43:44.369044 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.368611    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" event={"ID":"d722b25f-2e16-46c8-90f9-f3e5e3455e58","Type":"ContainerDied","Data":"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"}
Jul 24 13:43:44.369044 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.368692    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55fd855cd4-jwf8v" event={"ID":"d722b25f-2e16-46c8-90f9-f3e5e3455e58","Type":"ContainerDied","Data":"28a069c2fe4ace28ff37b6820625d14afe9c52b290215187f291cacb6641a05e"}
Jul 24 13:43:44.369044 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.368713    2527 scope.go:117] "RemoveContainer" containerID="444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"
Jul 24 13:43:44.378867 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.378848    2527 scope.go:117] "RemoveContainer" containerID="444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"
Jul 24 13:43:44.379197 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:43:44.379177    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9\": container with ID starting with 444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9 not found: ID does not exist" containerID="444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"
Jul 24 13:43:44.379250 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.379207    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9"} err="failed to get container status \"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9\": rpc error: code = NotFound desc = could not find container \"444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9\": container with ID starting with 444b8ae2ca351a1425706c51d4caded9783df0090b4a99d258b4cd55408224a9 not found: ID does not exist"
Jul 24 13:43:44.389943 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.389908    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:44.394214 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.394181    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55fd855cd4-jwf8v"]
Jul 24 13:43:44.513590 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:44.513562    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" path="/var/lib/kubelet/pods/d722b25f-2e16-46c8-90f9-f3e5e3455e58/volumes"
Jul 24 13:43:56.378686 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:56.378648    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:43:56.384958 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:43:56.384931    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:44:04.631143 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.631107    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b47444556-zp75g"]
Jul 24 13:44:04.631556 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.631450    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" containerName="manager"
Jul 24 13:44:04.631556 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.631462    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" containerName="manager"
Jul 24 13:44:04.631556 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.631534    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="d722b25f-2e16-46c8-90f9-f3e5e3455e58" containerName="manager"
Jul 24 13:44:04.635925 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.635906    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.638313 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.638290    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 13:44:04.638313 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.638303    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5cq4l\""
Jul 24 13:44:04.644363 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.644337    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b47444556-zp75g"]
Jul 24 13:44:04.688293 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.688252    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7a243d1-0d76-4f49-8591-387de9f80293-maas-api-tls\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.688502 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.688305    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a243d1-0d76-4f49-8591-387de9f80293-openshift-service-ca\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.688502 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.688333    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86rm\" (UniqueName: \"kubernetes.io/projected/a7a243d1-0d76-4f49-8591-387de9f80293-kube-api-access-q86rm\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.788823 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.788780    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86rm\" (UniqueName: \"kubernetes.io/projected/a7a243d1-0d76-4f49-8591-387de9f80293-kube-api-access-q86rm\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.789029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.788892    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7a243d1-0d76-4f49-8591-387de9f80293-maas-api-tls\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.789029 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.788933    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a243d1-0d76-4f49-8591-387de9f80293-openshift-service-ca\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.789589 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.789566    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a243d1-0d76-4f49-8591-387de9f80293-openshift-service-ca\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.791498 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.791478    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7a243d1-0d76-4f49-8591-387de9f80293-maas-api-tls\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.797924 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.797897    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86rm\" (UniqueName: \"kubernetes.io/projected/a7a243d1-0d76-4f49-8591-387de9f80293-kube-api-access-q86rm\") pod \"maas-api-b47444556-zp75g\" (UID: \"a7a243d1-0d76-4f49-8591-387de9f80293\") " pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:04.947154 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:04.947062    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:05.105874 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.105839    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b47444556-zp75g"]
Jul 24 13:44:05.111032 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:44:05.109932    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a243d1_0d76_4f49_8591_387de9f80293.slice/crio-e62abaee11fa89978d262e4dbd3954549a1eec2375306c0528d94f2d6dd661f5 WatchSource:0}: Error finding container e62abaee11fa89978d262e4dbd3954549a1eec2375306c0528d94f2d6dd661f5: Status 404 returned error can't find the container with id e62abaee11fa89978d262e4dbd3954549a1eec2375306c0528d94f2d6dd661f5
Jul 24 13:44:05.203797 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.203713    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ghdbf"]
Jul 24 13:44:05.208565 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.208546    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.211301 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.211283    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 13:44:05.215569 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.215541    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ghdbf"]
Jul 24 13:44:05.293253 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.293213    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tmn\" (UniqueName: \"kubernetes.io/projected/70aae07d-b447-4198-9132-48c66e275e68-kube-api-access-77tmn\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.293425 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.293328    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aae07d-b447-4198-9132-48c66e275e68-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.298130 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.298103    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"]
Jul 24 13:44:05.301515 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.301499    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.306670 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.306648    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pkx6x\""
Jul 24 13:44:05.317617 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.317590    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"]
Jul 24 13:44:05.393818 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.393783    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tmn\" (UniqueName: \"kubernetes.io/projected/70aae07d-b447-4198-9132-48c66e275e68-kube-api-access-77tmn\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.393985 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.393825    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59s6\" (UniqueName: \"kubernetes.io/projected/021dd046-1cdc-4904-9e26-a0c95efb9ace-kube-api-access-w59s6\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.393985 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.393847    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021dd046-1cdc-4904-9e26-a0c95efb9ace-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.394063 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.394001    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aae07d-b447-4198-9132-48c66e275e68-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.394722 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.394704    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aae07d-b447-4198-9132-48c66e275e68-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.406198 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.406163    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tmn\" (UniqueName: \"kubernetes.io/projected/70aae07d-b447-4198-9132-48c66e275e68-kube-api-access-77tmn\") pod \"payload-processing-5cc6cd444b-ghdbf\" (UID: \"70aae07d-b447-4198-9132-48c66e275e68\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.463880 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.463781    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g" event={"ID":"a7a243d1-0d76-4f49-8591-387de9f80293","Type":"ContainerStarted","Data":"e62abaee11fa89978d262e4dbd3954549a1eec2375306c0528d94f2d6dd661f5"}
Jul 24 13:44:05.495390 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.495351    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59s6\" (UniqueName: \"kubernetes.io/projected/021dd046-1cdc-4904-9e26-a0c95efb9ace-kube-api-access-w59s6\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.495390 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.495388    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021dd046-1cdc-4904-9e26-a0c95efb9ace-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.496135 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.496114    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021dd046-1cdc-4904-9e26-a0c95efb9ace-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.505109 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.505067    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59s6\" (UniqueName: \"kubernetes.io/projected/021dd046-1cdc-4904-9e26-a0c95efb9ace-kube-api-access-w59s6\") pod \"payload-pre-processing-687dbb6cf6-r26kw\" (UID: \"021dd046-1cdc-4904-9e26-a0c95efb9ace\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.520274 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.520231    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:05.611840 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.611803    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:05.667379 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.667349    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ghdbf"]
Jul 24 13:44:05.670493 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:44:05.670304    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70aae07d_b447_4198_9132_48c66e275e68.slice/crio-439788734d89678ff122b23a83039c840f3def7b4dea356c37ad6759baf3e9f9 WatchSource:0}: Error finding container 439788734d89678ff122b23a83039c840f3def7b4dea356c37ad6759baf3e9f9: Status 404 returned error can't find the container with id 439788734d89678ff122b23a83039c840f3def7b4dea356c37ad6759baf3e9f9
Jul 24 13:44:05.769414 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:05.769385    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"]
Jul 24 13:44:05.771754 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:44:05.771729    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021dd046_1cdc_4904_9e26_a0c95efb9ace.slice/crio-cf9aeff597d9cbade9fbb258b59b58b19d28e1e3f97d0eddd03ae035cfec837d WatchSource:0}: Error finding container cf9aeff597d9cbade9fbb258b59b58b19d28e1e3f97d0eddd03ae035cfec837d: Status 404 returned error can't find the container with id cf9aeff597d9cbade9fbb258b59b58b19d28e1e3f97d0eddd03ae035cfec837d
Jul 24 13:44:06.475384 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:06.475349    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw" event={"ID":"021dd046-1cdc-4904-9e26-a0c95efb9ace","Type":"ContainerStarted","Data":"cf9aeff597d9cbade9fbb258b59b58b19d28e1e3f97d0eddd03ae035cfec837d"}
Jul 24 13:44:06.477752 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:06.477718    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf" event={"ID":"70aae07d-b447-4198-9132-48c66e275e68","Type":"ContainerStarted","Data":"439788734d89678ff122b23a83039c840f3def7b4dea356c37ad6759baf3e9f9"}
Jul 24 13:44:08.490108 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.490074    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g" event={"ID":"a7a243d1-0d76-4f49-8591-387de9f80293","Type":"ContainerStarted","Data":"170380b6123dd7ad05f8f2f8403297d0003e6c126a1f5d5e5fda7f8627095b03"}
Jul 24 13:44:08.490539 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.490140    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:08.491836 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.491802    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw" event={"ID":"021dd046-1cdc-4904-9e26-a0c95efb9ace","Type":"ContainerStarted","Data":"16fb3b8326dce9da1495228d048c010e910d72d159174cd49799c75fc3a318ef"}
Jul 24 13:44:08.491995 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.491903    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:08.493725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.493699    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf" event={"ID":"70aae07d-b447-4198-9132-48c66e275e68","Type":"ContainerStarted","Data":"514b8ec79e86c6a2597fad238ca6ca45da1ed64aaa9d771856e775fe19315a67"}
Jul 24 13:44:08.493862 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.493798    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:08.528985 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.528881    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf" podStartSLOduration=0.817762236 podStartE2EDuration="3.528859445s" totalImagesPullingTime="2.711097209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:08.527131446 +0000 UTC m=+612.569179362" watchObservedRunningTime="2026-07-24 13:44:08.528859445 +0000 UTC m=+612.570907331"
Jul 24 13:44:08.529273 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.529233    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g" podStartSLOduration=1.326607013 podStartE2EDuration="4.529220466s" totalImagesPullingTime="3.202613453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:08.509566973 +0000 UTC m=+612.551614859" watchObservedRunningTime="2026-07-24 13:44:08.529220466 +0000 UTC m=+612.571268354"
Jul 24 13:44:08.547488 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:08.547413    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw" podStartSLOduration=0.931205758 podStartE2EDuration="3.547391949s" totalImagesPullingTime="2.616186191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:08.545683173 +0000 UTC m=+612.587731072" watchObservedRunningTime="2026-07-24 13:44:08.547391949 +0000 UTC m=+612.589439834"
Jul 24 13:44:14.505178 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:14.505144    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b47444556-zp75g"
Jul 24 13:44:19.501418 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:19.501333    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ghdbf"
Jul 24 13:44:19.501418 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:19.501412    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r26kw"
Jul 24 13:44:24.039573 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.039544    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"]
Jul 24 13:44:24.043039 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.043014    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.047194 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.047171    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2w2kd\""
Jul 24 13:44:24.047194 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.047179    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 13:44:24.047462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.047422    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 24 13:44:24.047462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.047428    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 13:44:24.061803 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.061773    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"]
Jul 24 13:44:24.170673 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170633    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.170849 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170687    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413017cd-cafd-459f-b573-9dff103a305b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.170849 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170814    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.170849 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170844    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.170970 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170874    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gvk\" (UniqueName: \"kubernetes.io/projected/413017cd-cafd-459f-b573-9dff103a305b-kube-api-access-p2gvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.170970 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.170918    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272116 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272080    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272116 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272115    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272134    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gvk\" (UniqueName: \"kubernetes.io/projected/413017cd-cafd-459f-b573-9dff103a305b-kube-api-access-p2gvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272154    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272176    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272354 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272205    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413017cd-cafd-459f-b573-9dff103a305b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272640 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272615    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272640 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272629    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.272768 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.272707    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.274859 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.274832    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413017cd-cafd-459f-b573-9dff103a305b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.274952 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.274934    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413017cd-cafd-459f-b573-9dff103a305b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.280839 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.280811    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gvk\" (UniqueName: \"kubernetes.io/projected/413017cd-cafd-459f-b573-9dff103a305b-kube-api-access-p2gvk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb\" (UID: \"413017cd-cafd-459f-b573-9dff103a305b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.354501 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.354463    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:24.514086 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.514061    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"]
Jul 24 13:44:24.516513 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:44:24.516487    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413017cd_cafd_459f_b573_9dff103a305b.slice/crio-c9a2ae8f1d175f8e73e9c745a147f01d7a23e4eb2f40ebd83cf176debd466cc6 WatchSource:0}: Error finding container c9a2ae8f1d175f8e73e9c745a147f01d7a23e4eb2f40ebd83cf176debd466cc6: Status 404 returned error can't find the container with id c9a2ae8f1d175f8e73e9c745a147f01d7a23e4eb2f40ebd83cf176debd466cc6
Jul 24 13:44:24.571135 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:24.571100    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb" event={"ID":"413017cd-cafd-459f-b573-9dff103a305b","Type":"ContainerStarted","Data":"c9a2ae8f1d175f8e73e9c745a147f01d7a23e4eb2f40ebd83cf176debd466cc6"}
Jul 24 13:44:26.583242 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:26.583208    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb" event={"ID":"413017cd-cafd-459f-b573-9dff103a305b","Type":"ContainerStarted","Data":"4f482d43c06f9fad63652b72d68bbd76513fc2bc628852a07f4a0b373fa462ac"}
Jul 24 13:44:26.601940 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:26.601882    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb" podStartSLOduration=1.4800928500000001 podStartE2EDuration="2.601863079s" totalImagesPullingTime="1.121770229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:26.600284757 +0000 UTC m=+630.642332643" watchObservedRunningTime="2026-07-24 13:44:26.601863079 +0000 UTC m=+630.643910964"
Jul 24 13:44:34.355006 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:34.354963    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:34.355643 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:34.355034    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:34.367728 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:34.367693    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:34.631404 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:34.631325    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb"
Jul 24 13:44:53.915943 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:53.915904    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"]
Jul 24 13:44:53.919249 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:53.919230    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:53.921173 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:53.921148    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 24 13:44:53.930907 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:53.930882    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"]
Jul 24 13:44:54.034299 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034252    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.034509 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034306    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.034509 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034369    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8a9808-abb2-492c-90ea-075928b05eba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.034509 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034414    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.034624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034517    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crdk\" (UniqueName: \"kubernetes.io/projected/3e8a9808-abb2-492c-90ea-075928b05eba-kube-api-access-5crdk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.034624 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.034554    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135255 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135213    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135495 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135266    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8a9808-abb2-492c-90ea-075928b05eba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135495 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135296    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135495 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135330    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crdk\" (UniqueName: \"kubernetes.io/projected/3e8a9808-abb2-492c-90ea-075928b05eba-kube-api-access-5crdk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135495 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135353    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135495 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135389    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135837 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135766    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135837 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135798    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.135913 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.135835    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.137726 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.137703    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8a9808-abb2-492c-90ea-075928b05eba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.138011 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.137990    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8a9808-abb2-492c-90ea-075928b05eba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.144346 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.144318    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crdk\" (UniqueName: \"kubernetes.io/projected/3e8a9808-abb2-492c-90ea-075928b05eba-kube-api-access-5crdk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z\" (UID: \"3e8a9808-abb2-492c-90ea-075928b05eba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.230265 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.230161    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:44:54.374506 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.374481    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"]
Jul 24 13:44:54.377341 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:44:54.377306    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8a9808_abb2_492c_90ea_075928b05eba.slice/crio-2bd2fd5ec022ae62c957debb896677773158141c98a11b818db4b1f5bcaff941 WatchSource:0}: Error finding container 2bd2fd5ec022ae62c957debb896677773158141c98a11b818db4b1f5bcaff941: Status 404 returned error can't find the container with id 2bd2fd5ec022ae62c957debb896677773158141c98a11b818db4b1f5bcaff941
Jul 24 13:44:54.714556 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.714518    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z" event={"ID":"3e8a9808-abb2-492c-90ea-075928b05eba","Type":"ContainerStarted","Data":"f473627c3eec797a0998b063051b2a24f94353424274060af72e55cbd0924dab"}
Jul 24 13:44:54.714720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.714582    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z" event={"ID":"3e8a9808-abb2-492c-90ea-075928b05eba","Type":"ContainerStarted","Data":"2bd2fd5ec022ae62c957debb896677773158141c98a11b818db4b1f5bcaff941"}
Jul 24 13:44:54.735115 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:44:54.735065    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z" podStartSLOduration=1.54834633 podStartE2EDuration="1.735049237s" totalImagesPullingTime="186.702907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:54.733752361 +0000 UTC m=+658.775800270" watchObservedRunningTime="2026-07-24 13:44:54.735049237 +0000 UTC m=+658.777097122"
Jul 24 13:45:04.231041 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:04.230999    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:45:04.231474 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:04.231064    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:45:04.243855 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:04.243825    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:45:04.772402 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:04.772371    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z"
Jul 24 13:45:22.289000 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.288960    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"]
Jul 24 13:45:22.292944 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.292911    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.295319 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.295294    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 13:45:22.299897 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.299870    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"]
Jul 24 13:45:22.398101 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398063    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxfk\" (UniqueName: \"kubernetes.io/projected/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-kube-api-access-qdxfk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.398300 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398123    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.398300 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398163    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.398300 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398258    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.398412 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398318    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.398412 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.398337    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499414 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499381    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499460    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499513    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499531    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499605 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499557    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxfk\" (UniqueName: \"kubernetes.io/projected/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-kube-api-access-qdxfk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499808 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499613    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499905 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499876    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499974 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499902    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.499974 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.499956    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.502200 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.502177    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.502373 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.502356    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.507369 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.507340    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxfk\" (UniqueName: \"kubernetes.io/projected/f5efa39f-a6a5-4fb3-912d-28b35ffadff6-kube-api-access-qdxfk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tbw54\" (UID: \"f5efa39f-a6a5-4fb3-912d-28b35ffadff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.603595 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.603546    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:22.749170 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.749140    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"]
Jul 24 13:45:22.751725 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:45:22.751696    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5efa39f_a6a5_4fb3_912d_28b35ffadff6.slice/crio-1ffc63dd8efbade7957635b9dc78a35af1f7eb4a0ed5df3dba42b9efbbe45cfd WatchSource:0}: Error finding container 1ffc63dd8efbade7957635b9dc78a35af1f7eb4a0ed5df3dba42b9efbbe45cfd: Status 404 returned error can't find the container with id 1ffc63dd8efbade7957635b9dc78a35af1f7eb4a0ed5df3dba42b9efbbe45cfd
Jul 24 13:45:22.753243 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.753224    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:45:22.846849 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:22.846809    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54" event={"ID":"f5efa39f-a6a5-4fb3-912d-28b35ffadff6","Type":"ContainerStarted","Data":"1ffc63dd8efbade7957635b9dc78a35af1f7eb4a0ed5df3dba42b9efbbe45cfd"}
Jul 24 13:45:23.853604 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:23.853565    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54" event={"ID":"f5efa39f-a6a5-4fb3-912d-28b35ffadff6","Type":"ContainerStarted","Data":"8684b0d1c609a2aca59b37fe3c8470a458e9e07f9c9108be9c8e8e7b429cc3c6"}
Jul 24 13:45:23.872149 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:23.872079    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54" podStartSLOduration=1.662518036 podStartE2EDuration="1.872062897s" totalImagesPullingTime="209.544861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:23.870261682 +0000 UTC m=+687.912309569" watchObservedRunningTime="2026-07-24 13:45:23.872062897 +0000 UTC m=+687.914111092"
Jul 24 13:45:32.604465 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:32.604391    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:32.604887 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:32.604493    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:32.617409 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:32.617383    2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:45:32.910516 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:45:32.910406    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tbw54"
Jul 24 13:47:18.775796 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:18.775693    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:47:18.776366 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:18.775916    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766d9c797c-fn4mg" podUID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" containerName="manager" containerID="cri-o://046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54" gracePeriod=30
Jul 24 13:47:19.037869 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.037802    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:47:19.089076 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.089033    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert\") pod \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") "
Jul 24 13:47:19.089275 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.089157    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdnx\" (UniqueName: \"kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx\") pod \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\" (UID: \"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0\") "
Jul 24 13:47:19.091401 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.091364    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx" (OuterVolumeSpecName: "kube-api-access-5wdnx") pod "c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" (UID: "c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0"). InnerVolumeSpecName "kube-api-access-5wdnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:47:19.091593 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.091421    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" (UID: "c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:47:19.190678 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.190633    2527 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-webhook-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:47:19.190678 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.190669    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdnx\" (UniqueName: \"kubernetes.io/projected/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0-kube-api-access-5wdnx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:47:19.412013 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.411979    2527 generic.go:356] "Generic (PLEG): container finished" podID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" containerID="046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54" exitCode=0
Jul 24 13:47:19.412215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.412026    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-fn4mg"
Jul 24 13:47:19.412215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.412057    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-fn4mg" event={"ID":"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0","Type":"ContainerDied","Data":"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"}
Jul 24 13:47:19.412215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.412135    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-fn4mg" event={"ID":"c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0","Type":"ContainerDied","Data":"532f81fbc2c00e226558c7e663b8c95d5e950c892d842efc98d57ed3c670ed78"}
Jul 24 13:47:19.412215 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.412165    2527 scope.go:117] "RemoveContainer" containerID="046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"
Jul 24 13:47:19.421600 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.421575    2527 scope.go:117] "RemoveContainer" containerID="046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"
Jul 24 13:47:19.421918 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:47:19.421897    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54\": container with ID starting with 046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54 not found: ID does not exist" containerID="046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"
Jul 24 13:47:19.421970 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.421928    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54"} err="failed to get container status \"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54\": rpc error: code = NotFound desc = could not find container \"046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54\": container with ID starting with 046130d9e648d1a5c225119b87f0b1933b5b06eb557997f1db04a56ab25a9c54 not found: ID does not exist"
Jul 24 13:47:19.433963 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.433924    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:47:19.437386 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:19.437359    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766d9c797c-fn4mg"]
Jul 24 13:47:20.513599 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:47:20.513570    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" path="/var/lib/kubelet/pods/c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0/volumes"
Jul 24 13:48:20.319372 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.319332    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:20.319866 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.319748    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" containerName="manager"
Jul 24 13:48:20.319866 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.319760    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" containerName="manager"
Jul 24 13:48:20.319866 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.319817    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f7eadc-2adb-4b6b-a29c-c2d16f9332b0" containerName="manager"
Jul 24 13:48:20.322704 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.322686    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.325612 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.325584    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 13:48:20.325742 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.325587    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mnksf\""
Jul 24 13:48:20.326240 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.326227    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 13:48:20.338469 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.338416    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:20.429254 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.429215    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vgp\" (UniqueName: \"kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.429462 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.429271    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.530480 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.530414    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vgp\" (UniqueName: \"kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.530690 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.530507    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.533221 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.533199    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.541725 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.541702    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vgp\" (UniqueName: \"kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp\") pod \"maas-controller-766d9c797c-dn4rn\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") " pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.632536 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.632499    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:20.778282 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:20.778245    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:20.779955 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:48:20.779924    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677fa8bf_afdd_4850_af67_299301efe94a.slice/crio-dcdf8a70ea570dce6d6e8b00690bc46e1674757e787b04591cb46beb03a3bac9 WatchSource:0}: Error finding container dcdf8a70ea570dce6d6e8b00690bc46e1674757e787b04591cb46beb03a3bac9: Status 404 returned error can't find the container with id dcdf8a70ea570dce6d6e8b00690bc46e1674757e787b04591cb46beb03a3bac9
Jul 24 13:48:21.710157 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:21.710111    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-dn4rn" event={"ID":"677fa8bf-afdd-4850-af67-299301efe94a","Type":"ContainerStarted","Data":"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"}
Jul 24 13:48:21.710645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:21.710175    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-dn4rn" event={"ID":"677fa8bf-afdd-4850-af67-299301efe94a","Type":"ContainerStarted","Data":"dcdf8a70ea570dce6d6e8b00690bc46e1674757e787b04591cb46beb03a3bac9"}
Jul 24 13:48:21.710645 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:21.710275    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:21.734508 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:21.734419    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766d9c797c-dn4rn" podStartSLOduration=1.334159512 podStartE2EDuration="1.734399285s" totalImagesPullingTime="400.239773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:21.732927897 +0000 UTC m=+865.774975783" watchObservedRunningTime="2026-07-24 13:48:21.734399285 +0000 UTC m=+865.776447170"
Jul 24 13:48:32.716618 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:32.716586    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:37.430328 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.430281    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:37.430757 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.430553    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766d9c797c-dn4rn" podUID="677fa8bf-afdd-4850-af67-299301efe94a" containerName="manager" containerID="cri-o://746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea" gracePeriod=30
Jul 24 13:48:37.684925 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.684854    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:37.784080 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.784047    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert\") pod \"677fa8bf-afdd-4850-af67-299301efe94a\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") "
Jul 24 13:48:37.784270 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.784165    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66vgp\" (UniqueName: \"kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp\") pod \"677fa8bf-afdd-4850-af67-299301efe94a\" (UID: \"677fa8bf-afdd-4850-af67-299301efe94a\") "
Jul 24 13:48:37.786527 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.786474    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "677fa8bf-afdd-4850-af67-299301efe94a" (UID: "677fa8bf-afdd-4850-af67-299301efe94a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:48:37.786670 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.786575    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp" (OuterVolumeSpecName: "kube-api-access-66vgp") pod "677fa8bf-afdd-4850-af67-299301efe94a" (UID: "677fa8bf-afdd-4850-af67-299301efe94a"). InnerVolumeSpecName "kube-api-access-66vgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:48:37.790283 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.790259    2527 generic.go:356] "Generic (PLEG): container finished" podID="677fa8bf-afdd-4850-af67-299301efe94a" containerID="746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea" exitCode=0
Jul 24 13:48:37.790410 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.790322    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-dn4rn"
Jul 24 13:48:37.790410 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.790336    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-dn4rn" event={"ID":"677fa8bf-afdd-4850-af67-299301efe94a","Type":"ContainerDied","Data":"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"}
Jul 24 13:48:37.790410 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.790391    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-dn4rn" event={"ID":"677fa8bf-afdd-4850-af67-299301efe94a","Type":"ContainerDied","Data":"dcdf8a70ea570dce6d6e8b00690bc46e1674757e787b04591cb46beb03a3bac9"}
Jul 24 13:48:37.790558 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.790414    2527 scope.go:117] "RemoveContainer" containerID="746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"
Jul 24 13:48:37.811757 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.810767    2527 scope.go:117] "RemoveContainer" containerID="746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"
Jul 24 13:48:37.811757 ip-10-0-129-199 kubenswrapper[2527]: E0724 13:48:37.811164    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea\": container with ID starting with 746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea not found: ID does not exist" containerID="746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"
Jul 24 13:48:37.811757 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.811196    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea"} err="failed to get container status \"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea\": rpc error: code = NotFound desc = could not find container \"746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea\": container with ID starting with 746aed73d2cb791804fa66010aede70a13973e5bd9c2b6e375a4f36b82375fea not found: ID does not exist"
Jul 24 13:48:37.824507 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.824479    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:37.828687 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.828658    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766d9c797c-dn4rn"]
Jul 24 13:48:37.885083 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.885040    2527 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/677fa8bf-afdd-4850-af67-299301efe94a-webhook-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:48:37.885083 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:37.885074    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66vgp\" (UniqueName: \"kubernetes.io/projected/677fa8bf-afdd-4850-af67-299301efe94a-kube-api-access-66vgp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 13:48:38.513642 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.513611    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677fa8bf-afdd-4850-af67-299301efe94a" path="/var/lib/kubelet/pods/677fa8bf-afdd-4850-af67-299301efe94a/volumes"
Jul 24 13:48:38.743943 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.743910    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766d9c797c-wj8fp"]
Jul 24 13:48:38.744316 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.744303    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677fa8bf-afdd-4850-af67-299301efe94a" containerName="manager"
Jul 24 13:48:38.744361 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.744317    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="677fa8bf-afdd-4850-af67-299301efe94a" containerName="manager"
Jul 24 13:48:38.744399 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.744394    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="677fa8bf-afdd-4850-af67-299301efe94a" containerName="manager"
Jul 24 13:48:38.748865 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.748846    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:38.752363 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.752343    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mnksf\""
Jul 24 13:48:38.752482 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.752345    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 13:48:38.752672 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.752657    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 13:48:38.760314 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.760286    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-wj8fp"]
Jul 24 13:48:38.896379 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.896348    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/975aabcd-e248-448b-a55c-82ef22695b11-webhook-cert\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:38.896586 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.896421    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7ls\" (UniqueName: \"kubernetes.io/projected/975aabcd-e248-448b-a55c-82ef22695b11-kube-api-access-hd7ls\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:38.997688 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.997638    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7ls\" (UniqueName: \"kubernetes.io/projected/975aabcd-e248-448b-a55c-82ef22695b11-kube-api-access-hd7ls\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:38.997862 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:38.997733    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/975aabcd-e248-448b-a55c-82ef22695b11-webhook-cert\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:39.000232 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.000213    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/975aabcd-e248-448b-a55c-82ef22695b11-webhook-cert\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:39.007061 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.007036    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7ls\" (UniqueName: \"kubernetes.io/projected/975aabcd-e248-448b-a55c-82ef22695b11-kube-api-access-hd7ls\") pod \"maas-controller-766d9c797c-wj8fp\" (UID: \"975aabcd-e248-448b-a55c-82ef22695b11\") " pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:39.060609 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.060571    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:39.207001 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.206973    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766d9c797c-wj8fp"]
Jul 24 13:48:39.209157 ip-10-0-129-199 kubenswrapper[2527]: W0724 13:48:39.209132    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975aabcd_e248_448b_a55c_82ef22695b11.slice/crio-5a76607c16974c6abec39de4b6566f7598930b0b8a64786722697fd947e78722 WatchSource:0}: Error finding container 5a76607c16974c6abec39de4b6566f7598930b0b8a64786722697fd947e78722: Status 404 returned error can't find the container with id 5a76607c16974c6abec39de4b6566f7598930b0b8a64786722697fd947e78722
Jul 24 13:48:39.803720 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.803658    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-wj8fp" event={"ID":"975aabcd-e248-448b-a55c-82ef22695b11","Type":"ContainerStarted","Data":"a3f5896502f13d021136f0fa1c9129abc87da7fee875c30056e062c766469182"}
Jul 24 13:48:39.804147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.803737    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766d9c797c-wj8fp" event={"ID":"975aabcd-e248-448b-a55c-82ef22695b11","Type":"ContainerStarted","Data":"5a76607c16974c6abec39de4b6566f7598930b0b8a64786722697fd947e78722"}
Jul 24 13:48:39.804147 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.803765    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:39.823551 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:39.823483    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766d9c797c-wj8fp" podStartSLOduration=1.3876093680000001 podStartE2EDuration="1.823461423s" totalImagesPullingTime="435.852055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:39.821365823 +0000 UTC m=+883.863413711" watchObservedRunningTime="2026-07-24 13:48:39.823461423 +0000 UTC m=+883.865509309"
Jul 24 13:48:50.812005 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:50.811925    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766d9c797c-wj8fp"
Jul 24 13:48:56.409156 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:56.409132    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:48:56.413663 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:48:56.413636    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:53:56.439643 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:53:56.439607    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:53:56.441857 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:53:56.441833    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:58:56.470417 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:58:56.470387    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 13:58:56.470968 ip-10-0-129-199 kubenswrapper[2527]: I0724 13:58:56.470807    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:01:15.516825 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.516738    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:15.520366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.520340    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.522763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.522740    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4dc6ea7f-serving-cert\""
Jul 24 14:01:15.532686 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.532649    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:15.671772 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.671728    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.671772 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.671776    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcflm\" (UniqueName: \"kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.671991 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.671809    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.773394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.773305    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.773394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.773358    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcflm\" (UniqueName: \"kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.773394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.773390    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.774040 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.774014    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.776175 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.776140    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.785566 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.785532    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcflm\" (UniqueName: \"kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm\") pod \"maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.830792 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.830755    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:15.987466 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.987367    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:15.989935 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:01:15.989902    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a819f9e_d55c_4f9b_a5c3_932e021c4fe4.slice/crio-d094bdf6ca9450d7f3ee8761b408b62ca4204bb3f324e1e5a128742ade4a1e97 WatchSource:0}: Error finding container d094bdf6ca9450d7f3ee8761b408b62ca4204bb3f324e1e5a128742ade4a1e97: Status 404 returned error can't find the container with id d094bdf6ca9450d7f3ee8761b408b62ca4204bb3f324e1e5a128742ade4a1e97
Jul 24 14:01:15.991243 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:15.991224    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:01:16.475943 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.475912    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" event={"ID":"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4","Type":"ContainerStarted","Data":"d094bdf6ca9450d7f3ee8761b408b62ca4204bb3f324e1e5a128742ade4a1e97"}
Jul 24 14:01:16.476092 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.476017    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:16.479309 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.479288    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.482315 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.482287    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4dc6ea7f\""
Jul 24 14:01:16.490716 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.490690    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:16.494299 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.494275    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.496497 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.496476    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4dc6ea7f-dockercfg-wvrb7\""
Jul 24 14:01:16.516653 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.516623    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:16.516808 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.516672    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:16.581060 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.581017    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2tr\" (UniqueName: \"kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.581526 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.581075    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.682290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.682195    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2tr\" (UniqueName: \"kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.682290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.682252    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.682290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.682275    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjwf\" (UniqueName: \"kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.682290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.682293    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.683062 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.683038    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.692568 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.692542    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2tr\" (UniqueName: \"kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr\") pod \"payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.783794 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.783756    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjwf\" (UniqueName: \"kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.783985 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.783824    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.784748 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.784726    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.791574 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.791547    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:16.794989 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.794968    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjwf\" (UniqueName: \"kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf\") pod \"payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") " pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.804754 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.804725    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:16.959973 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:16.959926    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:16.961905 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:01:16.961875    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a63dd_c413_4650_8727_c356af048ae7.slice/crio-ae543e7a6c8cdee70c7b57b29f38d2c83ca56d0cce063a41104aa449162c7ed2 WatchSource:0}: Error finding container ae543e7a6c8cdee70c7b57b29f38d2c83ca56d0cce063a41104aa449162c7ed2: Status 404 returned error can't find the container with id ae543e7a6c8cdee70c7b57b29f38d2c83ca56d0cce063a41104aa449162c7ed2
Jul 24 14:01:17.215646 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.215552    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:17.219151 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:01:17.219115    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67243d5e_e68f_45d6_b4cf_3fa2b12cbe5e.slice/crio-1d1e16a744d82d3825f1d79718a3a841962b26d5d41cc835839011ac4f7bea4f WatchSource:0}: Error finding container 1d1e16a744d82d3825f1d79718a3a841962b26d5d41cc835839011ac4f7bea4f: Status 404 returned error can't find the container with id 1d1e16a744d82d3825f1d79718a3a841962b26d5d41cc835839011ac4f7bea4f
Jul 24 14:01:17.482403 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.482285    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" event={"ID":"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4","Type":"ContainerStarted","Data":"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"}
Jul 24 14:01:17.482586 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.482470    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:17.483932 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.483908    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" event={"ID":"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e","Type":"ContainerStarted","Data":"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"}
Jul 24 14:01:17.484039 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.483946    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" event={"ID":"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e","Type":"ContainerStarted","Data":"1d1e16a744d82d3825f1d79718a3a841962b26d5d41cc835839011ac4f7bea4f"}
Jul 24 14:01:17.484039 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.483978    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:17.485400 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.485375    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" event={"ID":"1c0a63dd-c413-4650-8727-c356af048ae7","Type":"ContainerStarted","Data":"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"}
Jul 24 14:01:17.485560 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.485421    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" event={"ID":"1c0a63dd-c413-4650-8727-c356af048ae7","Type":"ContainerStarted","Data":"ae543e7a6c8cdee70c7b57b29f38d2c83ca56d0cce063a41104aa449162c7ed2"}
Jul 24 14:01:17.485560 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.485497    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:17.511942 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.511873    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" podStartSLOduration=2.081712316 podStartE2EDuration="2.511848765s" totalImagesPullingTime="430.136449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:01:17.5105814 +0000 UTC m=+1641.552629327" watchObservedRunningTime="2026-07-24 14:01:17.511848765 +0000 UTC m=+1641.553896654"
Jul 24 14:01:17.542895 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.542844    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" podStartSLOduration=1.542825223 podStartE2EDuration="1.542825223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:01:17.541844891 +0000 UTC m=+1641.583892777" watchObservedRunningTime="2026-07-24 14:01:17.542825223 +0000 UTC m=+1641.584873108"
Jul 24 14:01:17.561164 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:17.561111    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" podStartSLOduration=1.5610929040000001 podStartE2EDuration="1.561092904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:01:17.560382308 +0000 UTC m=+1641.602430206" watchObservedRunningTime="2026-07-24 14:01:17.561092904 +0000 UTC m=+1641.603140789"
Jul 24 14:01:23.495950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:23.495836    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:28.491916 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:28.491882    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:28.492334 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:28.491971    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:51.865723 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.865688    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:51.866264 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.865925    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" podUID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" containerName="maas-api" containerID="cri-o://1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802" gracePeriod=30
Jul 24 14:01:51.978120 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.978083    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:51.978415 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.978326    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" podUID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" containerName="payload-processing" containerID="cri-o://bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef" gracePeriod=30
Jul 24 14:01:51.984349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.984318    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:51.984652 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:51.984621    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" podUID="1c0a63dd-c413-4650-8727-c356af048ae7" containerName="payload-pre-processing" containerID="cri-o://acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2" gracePeriod=30
Jul 24 14:01:52.163355 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.163327    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:52.264712 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.264689    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:52.267710 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.267686    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:52.306497 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.306465    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcflm\" (UniqueName: \"kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm\") pod \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") "
Jul 24 14:01:52.306780 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.306752    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls\") pod \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") "
Jul 24 14:01:52.306915 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.306852    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca\") pod \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\" (UID: \"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4\") "
Jul 24 14:01:52.307868 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.307838    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" (UID: "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:01:52.309132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.309102    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm" (OuterVolumeSpecName: "kube-api-access-fcflm") pod "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" (UID: "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4"). InnerVolumeSpecName "kube-api-access-fcflm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:01:52.309220 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.309181    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" (UID: "6a819f9e-d55c-4f9b-a5c3-932e021c4fe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:01:52.408366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408276    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbjwf\" (UniqueName: \"kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf\") pod \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") "
Jul 24 14:01:52.408549 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408399    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume\") pod \"1c0a63dd-c413-4650-8727-c356af048ae7\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") "
Jul 24 14:01:52.408549 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408427    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume\") pod \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\" (UID: \"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e\") "
Jul 24 14:01:52.408549 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408514    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss2tr\" (UniqueName: \"kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr\") pod \"1c0a63dd-c413-4650-8727-c356af048ae7\" (UID: \"1c0a63dd-c413-4650-8727-c356af048ae7\") "
Jul 24 14:01:52.408776 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408758    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcflm\" (UniqueName: \"kubernetes.io/projected/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-kube-api-access-fcflm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.408848 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408784    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.408848 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408795    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.408848 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408762    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c0a63dd-c413-4650-8727-c356af048ae7" (UID: "1c0a63dd-c413-4650-8727-c356af048ae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:01:52.408963 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.408859    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" (UID: "67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:01:52.410824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.410799    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr" (OuterVolumeSpecName: "kube-api-access-ss2tr") pod "1c0a63dd-c413-4650-8727-c356af048ae7" (UID: "1c0a63dd-c413-4650-8727-c356af048ae7"). InnerVolumeSpecName "kube-api-access-ss2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:01:52.410824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.410817    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf" (OuterVolumeSpecName: "kube-api-access-dbjwf") pod "67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" (UID: "67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e"). InnerVolumeSpecName "kube-api-access-dbjwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:01:52.509417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.509381    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.509417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.509413    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2tr\" (UniqueName: \"kubernetes.io/projected/1c0a63dd-c413-4650-8727-c356af048ae7-kube-api-access-ss2tr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.509417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.509426    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjwf\" (UniqueName: \"kubernetes.io/projected/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e-kube-api-access-dbjwf\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.509417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.509453    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a63dd-c413-4650-8727-c356af048ae7-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:01:52.663906 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.663817    2527 generic.go:356] "Generic (PLEG): container finished" podID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" containerID="bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef" exitCode=0
Jul 24 14:01:52.663906 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.663862    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"
Jul 24 14:01:52.664101 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.663910    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" event={"ID":"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e","Type":"ContainerDied","Data":"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"}
Jul 24 14:01:52.664101 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.663972    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd" event={"ID":"67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e","Type":"ContainerDied","Data":"1d1e16a744d82d3825f1d79718a3a841962b26d5d41cc835839011ac4f7bea4f"}
Jul 24 14:01:52.664101 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.664004    2527 scope.go:117] "RemoveContainer" containerID="bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"
Jul 24 14:01:52.665303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.665278    2527 generic.go:356] "Generic (PLEG): container finished" podID="1c0a63dd-c413-4650-8727-c356af048ae7" containerID="acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2" exitCode=0
Jul 24 14:01:52.665410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.665348    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"
Jul 24 14:01:52.665410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.665366    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" event={"ID":"1c0a63dd-c413-4650-8727-c356af048ae7","Type":"ContainerDied","Data":"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"}
Jul 24 14:01:52.665525 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.665415    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf" event={"ID":"1c0a63dd-c413-4650-8727-c356af048ae7","Type":"ContainerDied","Data":"ae543e7a6c8cdee70c7b57b29f38d2c83ca56d0cce063a41104aa449162c7ed2"}
Jul 24 14:01:52.667260 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.667236    2527 generic.go:356] "Generic (PLEG): container finished" podID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" containerID="1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802" exitCode=0
Jul 24 14:01:52.667402 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.667276    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"
Jul 24 14:01:52.667402 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.667275    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" event={"ID":"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4","Type":"ContainerDied","Data":"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"}
Jul 24 14:01:52.667402 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.667386    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54" event={"ID":"6a819f9e-d55c-4f9b-a5c3-932e021c4fe4","Type":"ContainerDied","Data":"d094bdf6ca9450d7f3ee8761b408b62ca4204bb3f324e1e5a128742ade4a1e97"}
Jul 24 14:01:52.674465 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.674428    2527 scope.go:117] "RemoveContainer" containerID="bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"
Jul 24 14:01:52.674759 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:01:52.674738    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef\": container with ID starting with bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef not found: ID does not exist" containerID="bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"
Jul 24 14:01:52.674803 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.674772    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef"} err="failed to get container status \"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef\": rpc error: code = NotFound desc = could not find container \"bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef\": container with ID starting with bb155cb7ff82a2ce984df467a8d74ece1464528217cfea8cbe0d44748287a3ef not found: ID does not exist"
Jul 24 14:01:52.674803 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.674790    2527 scope.go:117] "RemoveContainer" containerID="acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"
Jul 24 14:01:52.683461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.683423    2527 scope.go:117] "RemoveContainer" containerID="acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"
Jul 24 14:01:52.683779 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:01:52.683761    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2\": container with ID starting with acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2 not found: ID does not exist" containerID="acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"
Jul 24 14:01:52.683834 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.683788    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2"} err="failed to get container status \"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2\": rpc error: code = NotFound desc = could not find container \"acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2\": container with ID starting with acfb956fe0bfa2cb8602d0f367e546acdab1a7ca3d363cb3bd5c46ccecb9e2d2 not found: ID does not exist"
Jul 24 14:01:52.683834 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.683809    2527 scope.go:117] "RemoveContainer" containerID="1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"
Jul 24 14:01:52.683903 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.683856    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:52.689158 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.689112    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4dc6ea7f-6df5788556-p4pbd"]
Jul 24 14:01:52.692734 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.692717    2527 scope.go:117] "RemoveContainer" containerID="1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"
Jul 24 14:01:52.693019 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:01:52.692999    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802\": container with ID starting with 1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802 not found: ID does not exist" containerID="1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"
Jul 24 14:01:52.693083 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.693025    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802"} err="failed to get container status \"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802\": rpc error: code = NotFound desc = could not find container \"1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802\": container with ID starting with 1eb93eb41990502f4d1af329d787d62a21003c45a6d5deb1db88404140d6d802 not found: ID does not exist"
Jul 24 14:01:52.701822 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.701792    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:52.706426 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.706399    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dc6ea7f-85c8cf4f4f-tpk54"]
Jul 24 14:01:52.722939 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.718200    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:52.725561 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:52.725527    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4dc6ea7f-59b449cbdd-hggqf"]
Jul 24 14:01:54.514203 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:54.514169    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0a63dd-c413-4650-8727-c356af048ae7" path="/var/lib/kubelet/pods/1c0a63dd-c413-4650-8727-c356af048ae7/volumes"
Jul 24 14:01:54.514603 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:54.514563    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" path="/var/lib/kubelet/pods/67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e/volumes"
Jul 24 14:01:54.514939 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:01:54.514926    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" path="/var/lib/kubelet/pods/6a819f9e-d55c-4f9b-a5c3-932e021c4fe4/volumes"
Jul 24 14:02:00.895947 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.895907    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:00.896505 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896487    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0a63dd-c413-4650-8727-c356af048ae7" containerName="payload-pre-processing"
Jul 24 14:02:00.896559 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896508    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0a63dd-c413-4650-8727-c356af048ae7" containerName="payload-pre-processing"
Jul 24 14:02:00.896559 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896528    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" containerName="payload-processing"
Jul 24 14:02:00.896559 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896538    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" containerName="payload-processing"
Jul 24 14:02:00.896655 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896560    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" containerName="maas-api"
Jul 24 14:02:00.896655 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896569    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" containerName="maas-api"
Jul 24 14:02:00.896713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896668    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a819f9e-d55c-4f9b-a5c3-932e021c4fe4" containerName="maas-api"
Jul 24 14:02:00.896713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896680    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="67243d5e-e68f-45d6-b4cf-3fa2b12cbe5e" containerName="payload-processing"
Jul 24 14:02:00.896713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.896692    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0a63dd-c413-4650-8727-c356af048ae7" containerName="payload-pre-processing"
Jul 24 14:02:00.901399 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.901375    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:00.903943 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.903896    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4a9d360c-serving-cert\""
Jul 24 14:02:00.910992 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.910967    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:00.987886 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.987847    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxh98\" (UniqueName: \"kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:00.988074 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.987932    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:00.988074 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:00.987982    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.089204 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.089160    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.089372 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.089227    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.089372 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.089278    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxh98\" (UniqueName: \"kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.089953 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.089886    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.091794 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.091774    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.100337 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.100314    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxh98\" (UniqueName: \"kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98\") pod \"maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.215250 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.215161    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:01.362145 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.362118    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:01.364623 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:01.364581    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43befd58_5c3f_48d4_9996_714248be7c88.slice/crio-6134752fe2dd45ec5dd22d61fdc9494ad9ee8d6af30967b9a0412b60a10db938 WatchSource:0}: Error finding container 6134752fe2dd45ec5dd22d61fdc9494ad9ee8d6af30967b9a0412b60a10db938: Status 404 returned error can't find the container with id 6134752fe2dd45ec5dd22d61fdc9494ad9ee8d6af30967b9a0412b60a10db938
Jul 24 14:02:01.704753 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.704720    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:01.709768 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.709746    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.714621 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.714586    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" event={"ID":"43befd58-5c3f-48d4-9996-714248be7c88","Type":"ContainerStarted","Data":"6134752fe2dd45ec5dd22d61fdc9494ad9ee8d6af30967b9a0412b60a10db938"}
Jul 24 14:02:01.716161 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.716138    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4a9d360c\""
Jul 24 14:02:01.716304 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.716256    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4a9d360c-dockercfg-q2vgn\""
Jul 24 14:02:01.758134 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.758105    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:01.789729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.789693    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:01.793767 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.793744    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:01.796461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.796405    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.796609 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.796513    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hdp\" (UniqueName: \"kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.808159 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.808126    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:01.897859 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.897809    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:01.898341 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.897882    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.898341 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.897923    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hdp\" (UniqueName: \"kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.898341 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.897940    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njrs4\" (UniqueName: \"kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:01.898690 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.898667    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.908754 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.908719    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hdp\" (UniqueName: \"kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp\") pod \"payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:01.999062 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.998914    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njrs4\" (UniqueName: \"kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:01.999212 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.999075    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:01.999798 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:01.999779    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:02.008175 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.008145    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njrs4\" (UniqueName: \"kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4\") pod \"payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:02.020969 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.020937    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:02.104154 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.104118    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:02.183212 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.183180    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:02.186500 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:02.186464    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aec35a8_6261_416c_a3fa_d930e51f4625.slice/crio-751493c63d0185984df43124d48ac4468aa8ac1cab186cfb1e2be2e9a2e5ab62 WatchSource:0}: Error finding container 751493c63d0185984df43124d48ac4468aa8ac1cab186cfb1e2be2e9a2e5ab62: Status 404 returned error can't find the container with id 751493c63d0185984df43124d48ac4468aa8ac1cab186cfb1e2be2e9a2e5ab62
Jul 24 14:02:02.499741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.499712    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:02.501861 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:02.501835    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a376520_664b_4619_aecc_d5548921b29a.slice/crio-c7e596bc1538c37f91ac847e9e2c453da3f350d944392395c5e847e284d6a2df WatchSource:0}: Error finding container c7e596bc1538c37f91ac847e9e2c453da3f350d944392395c5e847e284d6a2df: Status 404 returned error can't find the container with id c7e596bc1538c37f91ac847e9e2c453da3f350d944392395c5e847e284d6a2df
Jul 24 14:02:02.721533 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.721498    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" event={"ID":"5a376520-664b-4619-aecc-d5548921b29a","Type":"ContainerStarted","Data":"4b1a9222166d108e3f07a53afeb48425156cee2e98f53db89fb1bb7180356016"}
Jul 24 14:02:02.721737 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.721557    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" event={"ID":"5a376520-664b-4619-aecc-d5548921b29a","Type":"ContainerStarted","Data":"c7e596bc1538c37f91ac847e9e2c453da3f350d944392395c5e847e284d6a2df"}
Jul 24 14:02:02.721737 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.721674    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:02.723366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.723335    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" event={"ID":"7aec35a8-6261-416c-a3fa-d930e51f4625","Type":"ContainerStarted","Data":"88e2bd8f464ef58feaf2ca2ad2ed82737a8ebe967087588be1fb42b89d1a5fe1"}
Jul 24 14:02:02.723520 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.723389    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" event={"ID":"7aec35a8-6261-416c-a3fa-d930e51f4625","Type":"ContainerStarted","Data":"751493c63d0185984df43124d48ac4468aa8ac1cab186cfb1e2be2e9a2e5ab62"}
Jul 24 14:02:02.723520 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.723416    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:02.724992 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.724969    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" event={"ID":"43befd58-5c3f-48d4-9996-714248be7c88","Type":"ContainerStarted","Data":"8b086e26d0cf1370a9ada332e6d561d2a14367b344cf0ed2b0f3c8047c0383bb"}
Jul 24 14:02:02.725125 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.725112    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:02.741405 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.741354    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" podStartSLOduration=1.741336609 podStartE2EDuration="1.741336609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:02.739353308 +0000 UTC m=+1686.781401207" watchObservedRunningTime="2026-07-24 14:02:02.741336609 +0000 UTC m=+1686.783384494"
Jul 24 14:02:02.764827 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.764714    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" podStartSLOduration=2.315468665 podStartE2EDuration="2.764698156s" totalImagesPullingTime="449.229491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:02.762035401 +0000 UTC m=+1686.804083297" watchObservedRunningTime="2026-07-24 14:02:02.764698156 +0000 UTC m=+1686.806746040"
Jul 24 14:02:02.783383 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:02.783329    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" podStartSLOduration=1.7833161830000002 podStartE2EDuration="1.783316183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:02.781347877 +0000 UTC m=+1686.823395752" watchObservedRunningTime="2026-07-24 14:02:02.783316183 +0000 UTC m=+1686.825364134"
Jul 24 14:02:08.737611 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:08.737580    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:13.734152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:13.734121    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:13.734560 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:13.734295    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:36.687461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.687413    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:36.687947 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.687656    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" podUID="43befd58-5c3f-48d4-9996-714248be7c88" containerName="maas-api" containerID="cri-o://8b086e26d0cf1370a9ada332e6d561d2a14367b344cf0ed2b0f3c8047c0383bb" gracePeriod=30
Jul 24 14:02:36.754214 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.754177    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:36.754455 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.754407    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" podUID="5a376520-664b-4619-aecc-d5548921b29a" containerName="payload-processing" containerID="cri-o://4b1a9222166d108e3f07a53afeb48425156cee2e98f53db89fb1bb7180356016" gracePeriod=30
Jul 24 14:02:36.767376 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.766801    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:36.767376 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.767029    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" podUID="7aec35a8-6261-416c-a3fa-d930e51f4625" containerName="payload-pre-processing" containerID="cri-o://88e2bd8f464ef58feaf2ca2ad2ed82737a8ebe967087588be1fb42b89d1a5fe1" gracePeriod=30
Jul 24 14:02:36.897014 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.896975    2527 generic.go:356] "Generic (PLEG): container finished" podID="43befd58-5c3f-48d4-9996-714248be7c88" containerID="8b086e26d0cf1370a9ada332e6d561d2a14367b344cf0ed2b0f3c8047c0383bb" exitCode=0
Jul 24 14:02:36.897256 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.897043    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" event={"ID":"43befd58-5c3f-48d4-9996-714248be7c88","Type":"ContainerDied","Data":"8b086e26d0cf1370a9ada332e6d561d2a14367b344cf0ed2b0f3c8047c0383bb"}
Jul 24 14:02:36.899283 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.899254    2527 generic.go:356] "Generic (PLEG): container finished" podID="5a376520-664b-4619-aecc-d5548921b29a" containerID="4b1a9222166d108e3f07a53afeb48425156cee2e98f53db89fb1bb7180356016" exitCode=0
Jul 24 14:02:36.899465 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.899310    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" event={"ID":"5a376520-664b-4619-aecc-d5548921b29a","Type":"ContainerDied","Data":"4b1a9222166d108e3f07a53afeb48425156cee2e98f53db89fb1bb7180356016"}
Jul 24 14:02:36.901361 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.901333    2527 generic.go:356] "Generic (PLEG): container finished" podID="7aec35a8-6261-416c-a3fa-d930e51f4625" containerID="88e2bd8f464ef58feaf2ca2ad2ed82737a8ebe967087588be1fb42b89d1a5fe1" exitCode=0
Jul 24 14:02:36.901512 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:36.901393    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" event={"ID":"7aec35a8-6261-416c-a3fa-d930e51f4625","Type":"ContainerDied","Data":"88e2bd8f464ef58feaf2ca2ad2ed82737a8ebe967087588be1fb42b89d1a5fe1"}
Jul 24 14:02:37.057260 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.057235    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:37.064913 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.064892    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:37.069515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.069494    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:37.117621 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117588    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca\") pod \"43befd58-5c3f-48d4-9996-714248be7c88\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") "
Jul 24 14:02:37.117818 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117670    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume\") pod \"5a376520-664b-4619-aecc-d5548921b29a\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") "
Jul 24 14:02:37.117818 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117712    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls\") pod \"43befd58-5c3f-48d4-9996-714248be7c88\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") "
Jul 24 14:02:37.117818 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117768    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxh98\" (UniqueName: \"kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98\") pod \"43befd58-5c3f-48d4-9996-714248be7c88\" (UID: \"43befd58-5c3f-48d4-9996-714248be7c88\") "
Jul 24 14:02:37.117818 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117797    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njrs4\" (UniqueName: \"kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4\") pod \"5a376520-664b-4619-aecc-d5548921b29a\" (UID: \"5a376520-664b-4619-aecc-d5548921b29a\") "
Jul 24 14:02:37.118040 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117827    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume\") pod \"7aec35a8-6261-416c-a3fa-d930e51f4625\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") "
Jul 24 14:02:37.118040 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117853    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45hdp\" (UniqueName: \"kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp\") pod \"7aec35a8-6261-416c-a3fa-d930e51f4625\" (UID: \"7aec35a8-6261-416c-a3fa-d930e51f4625\") "
Jul 24 14:02:37.118040 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.117961    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43befd58-5c3f-48d4-9996-714248be7c88" (UID: "43befd58-5c3f-48d4-9996-714248be7c88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:02:37.118195 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.118044    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a376520-664b-4619-aecc-d5548921b29a" (UID: "5a376520-664b-4619-aecc-d5548921b29a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:02:37.118251 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.118208    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a376520-664b-4619-aecc-d5548921b29a-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.118396 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.118229    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43befd58-5c3f-48d4-9996-714248be7c88-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.118523 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.118482    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aec35a8-6261-416c-a3fa-d930e51f4625" (UID: "7aec35a8-6261-416c-a3fa-d930e51f4625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:02:37.120742 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.120646    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98" (OuterVolumeSpecName: "kube-api-access-wxh98") pod "43befd58-5c3f-48d4-9996-714248be7c88" (UID: "43befd58-5c3f-48d4-9996-714248be7c88"). InnerVolumeSpecName "kube-api-access-wxh98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:02:37.120742 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.120705    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43befd58-5c3f-48d4-9996-714248be7c88" (UID: "43befd58-5c3f-48d4-9996-714248be7c88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:02:37.120742 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.120733    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4" (OuterVolumeSpecName: "kube-api-access-njrs4") pod "5a376520-664b-4619-aecc-d5548921b29a" (UID: "5a376520-664b-4619-aecc-d5548921b29a"). InnerVolumeSpecName "kube-api-access-njrs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:02:37.120981 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.120762    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp" (OuterVolumeSpecName: "kube-api-access-45hdp") pod "7aec35a8-6261-416c-a3fa-d930e51f4625" (UID: "7aec35a8-6261-416c-a3fa-d930e51f4625"). InnerVolumeSpecName "kube-api-access-45hdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:02:37.219031 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.218986    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43befd58-5c3f-48d4-9996-714248be7c88-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.219031 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.219019    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxh98\" (UniqueName: \"kubernetes.io/projected/43befd58-5c3f-48d4-9996-714248be7c88-kube-api-access-wxh98\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.219031 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.219030    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njrs4\" (UniqueName: \"kubernetes.io/projected/5a376520-664b-4619-aecc-d5548921b29a-kube-api-access-njrs4\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.219031 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.219039    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aec35a8-6261-416c-a3fa-d930e51f4625-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.219396 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.219049    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hdp\" (UniqueName: \"kubernetes.io/projected/7aec35a8-6261-416c-a3fa-d930e51f4625-kube-api-access-45hdp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:02:37.907561 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.907526    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x" event={"ID":"5a376520-664b-4619-aecc-d5548921b29a","Type":"ContainerDied","Data":"c7e596bc1538c37f91ac847e9e2c453da3f350d944392395c5e847e284d6a2df"}
Jul 24 14:02:37.908032 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.907544    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"
Jul 24 14:02:37.908032 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.907594    2527 scope.go:117] "RemoveContainer" containerID="4b1a9222166d108e3f07a53afeb48425156cee2e98f53db89fb1bb7180356016"
Jul 24 14:02:37.909019 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.908994    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"
Jul 24 14:02:37.909311 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.908996    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q" event={"ID":"7aec35a8-6261-416c-a3fa-d930e51f4625","Type":"ContainerDied","Data":"751493c63d0185984df43124d48ac4468aa8ac1cab186cfb1e2be2e9a2e5ab62"}
Jul 24 14:02:37.910512 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.910479    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl" event={"ID":"43befd58-5c3f-48d4-9996-714248be7c88","Type":"ContainerDied","Data":"6134752fe2dd45ec5dd22d61fdc9494ad9ee8d6af30967b9a0412b60a10db938"}
Jul 24 14:02:37.910637 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.910483    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"
Jul 24 14:02:37.918389 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.918366    2527 scope.go:117] "RemoveContainer" containerID="88e2bd8f464ef58feaf2ca2ad2ed82737a8ebe967087588be1fb42b89d1a5fe1"
Jul 24 14:02:37.927160 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.927139    2527 scope.go:117] "RemoveContainer" containerID="8b086e26d0cf1370a9ada332e6d561d2a14367b344cf0ed2b0f3c8047c0383bb"
Jul 24 14:02:37.935334 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.935305    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:37.940566 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.940543    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4a9d360c-6646cb858f-p8d8x"]
Jul 24 14:02:37.958826 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.958753    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:37.964730 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.964698    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a9d360c-6bd97f9896-bmtjl"]
Jul 24 14:02:37.976529 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.976497    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:37.980857 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:37.980832    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a9d360c-6d74957ddd-tft6q"]
Jul 24 14:02:38.514719 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:38.514691    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43befd58-5c3f-48d4-9996-714248be7c88" path="/var/lib/kubelet/pods/43befd58-5c3f-48d4-9996-714248be7c88/volumes"
Jul 24 14:02:38.515112 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:38.515098    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a376520-664b-4619-aecc-d5548921b29a" path="/var/lib/kubelet/pods/5a376520-664b-4619-aecc-d5548921b29a/volumes"
Jul 24 14:02:38.515428 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:38.515416    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aec35a8-6261-416c-a3fa-d930e51f4625" path="/var/lib/kubelet/pods/7aec35a8-6261-416c-a3fa-d930e51f4625/volumes"
Jul 24 14:02:45.106081 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106049    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106447    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aec35a8-6261-416c-a3fa-d930e51f4625" containerName="payload-pre-processing"
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106461    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aec35a8-6261-416c-a3fa-d930e51f4625" containerName="payload-pre-processing"
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106473    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43befd58-5c3f-48d4-9996-714248be7c88" containerName="maas-api"
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106478    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="43befd58-5c3f-48d4-9996-714248be7c88" containerName="maas-api"
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106487    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a376520-664b-4619-aecc-d5548921b29a" containerName="payload-processing"
Jul 24 14:02:45.106506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106493    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a376520-664b-4619-aecc-d5548921b29a" containerName="payload-processing"
Jul 24 14:02:45.106729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106560    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="43befd58-5c3f-48d4-9996-714248be7c88" containerName="maas-api"
Jul 24 14:02:45.106729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106568    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a376520-664b-4619-aecc-d5548921b29a" containerName="payload-processing"
Jul 24 14:02:45.106729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.106573    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aec35a8-6261-416c-a3fa-d930e51f4625" containerName="payload-pre-processing"
Jul 24 14:02:45.111306 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.111287    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.114924 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.114895    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6538ff29-serving-cert\""
Jul 24 14:02:45.124300 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.124268    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:02:45.191200 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.191167    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.191378 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.191277    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.191378 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.191327    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wkrr\" (UniqueName: \"kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.292349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.292313    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wkrr\" (UniqueName: \"kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.292540 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.292361    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.292540 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.292422    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.293122 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.293095    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.294891 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.294871    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.318489 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.318460    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wkrr\" (UniqueName: \"kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr\") pod \"maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.421806 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.421708    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:45.580561 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.580530    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:02:45.582320 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:45.582280    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdf4805_c3f0_4c24_8fb8_57f36f5a06fc.slice/crio-6d7869f52156311c376f515d94e836c75a586436e4be55b03df65e2ba8a8a7ea WatchSource:0}: Error finding container 6d7869f52156311c376f515d94e836c75a586436e4be55b03df65e2ba8a8a7ea: Status 404 returned error can't find the container with id 6d7869f52156311c376f515d94e836c75a586436e4be55b03df65e2ba8a8a7ea
Jul 24 14:02:45.954491 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:45.954458    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" event={"ID":"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc","Type":"ContainerStarted","Data":"6d7869f52156311c376f515d94e836c75a586436e4be55b03df65e2ba8a8a7ea"}
Jul 24 14:02:46.747222 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.747185    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:02:46.750708 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.750688    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:46.753395 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.753371    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6538ff29\""
Jul 24 14:02:46.753536 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.753410    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6538ff29-dockercfg-7q7w7\""
Jul 24 14:02:46.775137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.775103    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:02:46.907839 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.907794    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdz6\" (UniqueName: \"kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:46.907839 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.907840    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:46.950516 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.950481    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:02:46.953973 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.953957    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:46.963189 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.963158    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" event={"ID":"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc","Type":"ContainerStarted","Data":"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"}
Jul 24 14:02:46.963343 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.963303    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:46.966560 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:46.966537    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:02:47.004351 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.004249    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" podStartSLOduration=1.662526183 podStartE2EDuration="2.004232875s" totalImagesPullingTime="341.706692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:47.001634181 +0000 UTC m=+1731.043682079" watchObservedRunningTime="2026-07-24 14:02:47.004232875 +0000 UTC m=+1731.046280759"
Jul 24 14:02:47.009194 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.009159    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdz6\" (UniqueName: \"kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:47.009264 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.009206    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:47.009903 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.009884    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:47.021676 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.021652    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdz6\" (UniqueName: \"kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6\") pod \"payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:47.061602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.061559    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:47.110050 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.110014    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqjx\" (UniqueName: \"kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.110224 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.110183    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.211048 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.211006    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.211223 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.211110    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqjx\" (UniqueName: \"kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.211789 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.211767    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.236216 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.236184    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqjx\" (UniqueName: \"kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx\") pod \"payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") " pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.252271 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.252246    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:02:47.255119 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:47.255048    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491274e_ef6b_43d6_a77c_06626bfa20cc.slice/crio-0d5363bb5e0a3dfb43138f01c3ffc1f9e90f74d277609ffe1a306e1c8ddd2beb WatchSource:0}: Error finding container 0d5363bb5e0a3dfb43138f01c3ffc1f9e90f74d277609ffe1a306e1c8ddd2beb: Status 404 returned error can't find the container with id 0d5363bb5e0a3dfb43138f01c3ffc1f9e90f74d277609ffe1a306e1c8ddd2beb
Jul 24 14:02:47.266658 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.266629    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.433402 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.433367    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:02:47.436644 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:02:47.436614    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1207f1_aa16_44cc_a0ec_082e05fb4a8e.slice/crio-6be2bef7601c4b7b9bde2a57b17db4196624096478484a1b45dc3f97a487dac6 WatchSource:0}: Error finding container 6be2bef7601c4b7b9bde2a57b17db4196624096478484a1b45dc3f97a487dac6: Status 404 returned error can't find the container with id 6be2bef7601c4b7b9bde2a57b17db4196624096478484a1b45dc3f97a487dac6
Jul 24 14:02:47.970570 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.970530    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" event={"ID":"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e","Type":"ContainerStarted","Data":"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"}
Jul 24 14:02:47.971043 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.970597    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" event={"ID":"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e","Type":"ContainerStarted","Data":"6be2bef7601c4b7b9bde2a57b17db4196624096478484a1b45dc3f97a487dac6"}
Jul 24 14:02:47.971043 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.970658    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:02:47.972250 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.972227    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" event={"ID":"a491274e-ef6b-43d6-a77c-06626bfa20cc","Type":"ContainerStarted","Data":"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"}
Jul 24 14:02:47.972355 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.972267    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" event={"ID":"a491274e-ef6b-43d6-a77c-06626bfa20cc","Type":"ContainerStarted","Data":"0d5363bb5e0a3dfb43138f01c3ffc1f9e90f74d277609ffe1a306e1c8ddd2beb"}
Jul 24 14:02:47.972445 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:47.972418    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:48.009284 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:48.009234    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" podStartSLOduration=2.009217324 podStartE2EDuration="2.009217324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:48.004869216 +0000 UTC m=+1732.046917114" watchObservedRunningTime="2026-07-24 14:02:48.009217324 +0000 UTC m=+1732.051265208"
Jul 24 14:02:48.042542 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:48.042481    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" podStartSLOduration=2.042466253 podStartE2EDuration="2.042466253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:48.040759069 +0000 UTC m=+1732.082806954" watchObservedRunningTime="2026-07-24 14:02:48.042466253 +0000 UTC m=+1732.084514137"
Jul 24 14:02:52.978278 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:52.978247    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:02:58.982652 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:58.982617    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:02:58.983081 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:02:58.982699    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:03:21.334904 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.334868    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:03:21.335403 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.335084    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" podUID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" containerName="maas-api" containerID="cri-o://d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac" gracePeriod=30
Jul 24 14:03:21.412999 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.412960    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:03:21.413258 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.413211    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" podUID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" containerName="payload-processing" containerID="cri-o://07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205" gracePeriod=30
Jul 24 14:03:21.432807 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.432778    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:03:21.433005 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.432982    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" podUID="a491274e-ef6b-43d6-a77c-06626bfa20cc" containerName="payload-pre-processing" containerID="cri-o://698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905" gracePeriod=30
Jul 24 14:03:21.703106 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.703074    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:03:21.739910 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.739863    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:03:21.753425 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.753401    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:03:21.816193 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816162    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume\") pod \"a491274e-ef6b-43d6-a77c-06626bfa20cc\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") "
Jul 24 14:03:21.816391 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816219    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dqjx\" (UniqueName: \"kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx\") pod \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") "
Jul 24 14:03:21.816391 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816260    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wkrr\" (UniqueName: \"kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr\") pod \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") "
Jul 24 14:03:21.816391 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816278    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca\") pod \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") "
Jul 24 14:03:21.816391 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816297    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls\") pod \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\" (UID: \"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc\") "
Jul 24 14:03:21.816391 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816354    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjdz6\" (UniqueName: \"kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6\") pod \"a491274e-ef6b-43d6-a77c-06626bfa20cc\" (UID: \"a491274e-ef6b-43d6-a77c-06626bfa20cc\") "
Jul 24 14:03:21.816756 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816411    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume\") pod \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\" (UID: \"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e\") "
Jul 24 14:03:21.816756 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816673    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a491274e-ef6b-43d6-a77c-06626bfa20cc" (UID: "a491274e-ef6b-43d6-a77c-06626bfa20cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:03:21.816756 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816707    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" (UID: "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:03:21.816918 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816781    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a491274e-ef6b-43d6-a77c-06626bfa20cc-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.816994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.816974    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" (UID: "1d1207f1-aa16-44cc-a0ec-082e05fb4a8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:03:21.819310 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.819282    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr" (OuterVolumeSpecName: "kube-api-access-7wkrr") pod "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" (UID: "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc"). InnerVolumeSpecName "kube-api-access-7wkrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:03:21.819419 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.819285    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" (UID: "0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:03:21.819419 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.819346    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx" (OuterVolumeSpecName: "kube-api-access-4dqjx") pod "1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" (UID: "1d1207f1-aa16-44cc-a0ec-082e05fb4a8e"). InnerVolumeSpecName "kube-api-access-4dqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:03:21.819419 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.819349    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6" (OuterVolumeSpecName: "kube-api-access-sjdz6") pod "a491274e-ef6b-43d6-a77c-06626bfa20cc" (UID: "a491274e-ef6b-43d6-a77c-06626bfa20cc"). InnerVolumeSpecName "kube-api-access-sjdz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917870    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wkrr\" (UniqueName: \"kubernetes.io/projected/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-kube-api-access-7wkrr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917902    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917912    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917921    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdz6\" (UniqueName: \"kubernetes.io/projected/a491274e-ef6b-43d6-a77c-06626bfa20cc-kube-api-access-sjdz6\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917930    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:21.917960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:21.917938    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqjx\" (UniqueName: \"kubernetes.io/projected/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e-kube-api-access-4dqjx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:03:22.145713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.145679    2527 generic.go:356] "Generic (PLEG): container finished" podID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" containerID="07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205" exitCode=0
Jul 24 14:03:22.145914 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.145733    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"
Jul 24 14:03:22.145914 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.145751    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" event={"ID":"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e","Type":"ContainerDied","Data":"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"}
Jul 24 14:03:22.145914 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.145803    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf" event={"ID":"1d1207f1-aa16-44cc-a0ec-082e05fb4a8e","Type":"ContainerDied","Data":"6be2bef7601c4b7b9bde2a57b17db4196624096478484a1b45dc3f97a487dac6"}
Jul 24 14:03:22.145914 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.145825    2527 scope.go:117] "RemoveContainer" containerID="07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"
Jul 24 14:03:22.147050 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.147027    2527 generic.go:356] "Generic (PLEG): container finished" podID="a491274e-ef6b-43d6-a77c-06626bfa20cc" containerID="698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905" exitCode=0
Jul 24 14:03:22.147132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.147064    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"
Jul 24 14:03:22.147132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.147114    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" event={"ID":"a491274e-ef6b-43d6-a77c-06626bfa20cc","Type":"ContainerDied","Data":"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"}
Jul 24 14:03:22.147259 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.147159    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v" event={"ID":"a491274e-ef6b-43d6-a77c-06626bfa20cc","Type":"ContainerDied","Data":"0d5363bb5e0a3dfb43138f01c3ffc1f9e90f74d277609ffe1a306e1c8ddd2beb"}
Jul 24 14:03:22.149015 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.148986    2527 generic.go:356] "Generic (PLEG): container finished" podID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" containerID="d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac" exitCode=0
Jul 24 14:03:22.149155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.149051    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"
Jul 24 14:03:22.149155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.149060    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" event={"ID":"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc","Type":"ContainerDied","Data":"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"}
Jul 24 14:03:22.149155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.149101    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97" event={"ID":"0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc","Type":"ContainerDied","Data":"6d7869f52156311c376f515d94e836c75a586436e4be55b03df65e2ba8a8a7ea"}
Jul 24 14:03:22.156706 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.156687    2527 scope.go:117] "RemoveContainer" containerID="07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"
Jul 24 14:03:22.157067 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:03:22.157039    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205\": container with ID starting with 07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205 not found: ID does not exist" containerID="07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"
Jul 24 14:03:22.157212 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.157081    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205"} err="failed to get container status \"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205\": rpc error: code = NotFound desc = could not find container \"07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205\": container with ID starting with 07c9328bf6abdbdc66799c863695af8e7cefc8ff1c9602602019c0d0042ba205 not found: ID does not exist"
Jul 24 14:03:22.157212 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.157129    2527 scope.go:117] "RemoveContainer" containerID="698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"
Jul 24 14:03:22.166411 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.166389    2527 scope.go:117] "RemoveContainer" containerID="698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"
Jul 24 14:03:22.166761 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:03:22.166742    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905\": container with ID starting with 698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905 not found: ID does not exist" containerID="698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"
Jul 24 14:03:22.166809 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.166768    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905"} err="failed to get container status \"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905\": rpc error: code = NotFound desc = could not find container \"698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905\": container with ID starting with 698b4cce7ba822c8d5a0631f52b58f7950119a0f98bf8d186c9deb9e7cbbc905 not found: ID does not exist"
Jul 24 14:03:22.166809 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.166785    2527 scope.go:117] "RemoveContainer" containerID="d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"
Jul 24 14:03:22.172883 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.172824    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:03:22.175310 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.175293    2527 scope.go:117] "RemoveContainer" containerID="d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"
Jul 24 14:03:22.175687 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:03:22.175662    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac\": container with ID starting with d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac not found: ID does not exist" containerID="d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"
Jul 24 14:03:22.175747 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.175693    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac"} err="failed to get container status \"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac\": rpc error: code = NotFound desc = could not find container \"d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac\": container with ID starting with d59c511e261023a1e6357779cdecdca5a75d6c3acfe4bc4ee15f057c3e2f28ac not found: ID does not exist"
Jul 24 14:03:22.178815 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.178796    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6538ff29-b8cd8c8bb-zsjvf"]
Jul 24 14:03:22.188547 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.188520    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:03:22.193035 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.193010    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6538ff29-6b8c4cfdd-hsr97"]
Jul 24 14:03:22.203321 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.203292    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:03:22.207100 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.207075    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6538ff29-7db4d5f5b4-jzs6v"]
Jul 24 14:03:22.513575 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.513498    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" path="/var/lib/kubelet/pods/0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc/volumes"
Jul 24 14:03:22.513935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.513878    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" path="/var/lib/kubelet/pods/1d1207f1-aa16-44cc-a0ec-082e05fb4a8e/volumes"
Jul 24 14:03:22.514201 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:22.514188    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a491274e-ef6b-43d6-a77c-06626bfa20cc" path="/var/lib/kubelet/pods/a491274e-ef6b-43d6-a77c-06626bfa20cc/volumes"
Jul 24 14:03:30.565637 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.565604    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.565989    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a491274e-ef6b-43d6-a77c-06626bfa20cc" containerName="payload-pre-processing"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566001    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="a491274e-ef6b-43d6-a77c-06626bfa20cc" containerName="payload-pre-processing"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566021    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" containerName="payload-processing"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566026    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" containerName="payload-processing"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566036    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" containerName="maas-api"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566044    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" containerName="maas-api"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566097    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1207f1-aa16-44cc-a0ec-082e05fb4a8e" containerName="payload-processing"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566106    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cdf4805-c3f0-4c24-8fb8-57f36f5a06fc" containerName="maas-api"
Jul 24 14:03:30.566137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.566116    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="a491274e-ef6b-43d6-a77c-06626bfa20cc" containerName="payload-pre-processing"
Jul 24 14:03:30.569147 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.569130    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.573446 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.573410    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9846c499-serving-cert\""
Jul 24 14:03:30.589613 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.589583    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:03:30.703382 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.703346    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.703382 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.703384    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zbz\" (UniqueName: \"kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.703635 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.703414    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.804871 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.804838    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.804871 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.804874    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zbz\" (UniqueName: \"kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.805160 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.804903    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.805607 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.805580    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.807481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.807463    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.816741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.816681    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zbz\" (UniqueName: \"kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz\") pod \"maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:30.880503 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:30.880464    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:31.027739 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.027715    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:03:31.029964 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:03:31.029935    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf617d5ec_8c0d_427e_9af6_37543d59cf22.slice/crio-6bec46487715c23b560fa4c6da4972c70e9503b70d520d14cddbf4000a2afb32 WatchSource:0}: Error finding container 6bec46487715c23b560fa4c6da4972c70e9503b70d520d14cddbf4000a2afb32: Status 404 returned error can't find the container with id 6bec46487715c23b560fa4c6da4972c70e9503b70d520d14cddbf4000a2afb32
Jul 24 14:03:31.196499 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.196459    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" event={"ID":"f617d5ec-8c0d-427e-9af6-37543d59cf22","Type":"ContainerStarted","Data":"6bec46487715c23b560fa4c6da4972c70e9503b70d520d14cddbf4000a2afb32"}
Jul 24 14:03:31.502368 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.502290    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:03:31.506260 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.506240    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.509535 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.509509    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9846c499\""
Jul 24 14:03:31.509635 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.509509    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9846c499-dockercfg-nmkln\""
Jul 24 14:03:31.526554 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.526479    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:03:31.589618 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.589586    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:03:31.592976 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.592955    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.610918 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.610889    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:03:31.611259 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.611236    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.611315 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.611281    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kct\" (UniqueName: \"kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.712764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.712725    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.712968 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.712793    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.712968 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.712850    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kct\" (UniqueName: \"kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.712968 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.712871    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2x2w\" (UniqueName: \"kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.713729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.713708    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.723041 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.723015    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kct\" (UniqueName: \"kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct\") pod \"payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.814202 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.814113    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.814202 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.814163    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2x2w\" (UniqueName: \"kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.814881 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.814859    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.818024 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.817999    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:31.824020 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.823998    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2x2w\" (UniqueName: \"kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w\") pod \"payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") " pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.905387 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.905350    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:31.965619 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:31.964769    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:03:31.969287 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:03:31.969247    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b519a71_ac2d_4849_a94c_fd3d48f88a71.slice/crio-291217e041081f0a88e87908d58cdc2d8f224e0292c647f0bc553d9ca9d6b7e4 WatchSource:0}: Error finding container 291217e041081f0a88e87908d58cdc2d8f224e0292c647f0bc553d9ca9d6b7e4: Status 404 returned error can't find the container with id 291217e041081f0a88e87908d58cdc2d8f224e0292c647f0bc553d9ca9d6b7e4
Jul 24 14:03:32.066350 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.066276    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:03:32.070867 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:03:32.070837    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194acd13_ab77_49eb_b176_071814be2b95.slice/crio-fa9dbc42e081eddafe8e577831f59d4c2722c5d928a182ba6e28bee6c8fd7822 WatchSource:0}: Error finding container fa9dbc42e081eddafe8e577831f59d4c2722c5d928a182ba6e28bee6c8fd7822: Status 404 returned error can't find the container with id fa9dbc42e081eddafe8e577831f59d4c2722c5d928a182ba6e28bee6c8fd7822
Jul 24 14:03:32.205338 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.205302    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" event={"ID":"9b519a71-ac2d-4849-a94c-fd3d48f88a71","Type":"ContainerStarted","Data":"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"}
Jul 24 14:03:32.205569 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.205367    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" event={"ID":"9b519a71-ac2d-4849-a94c-fd3d48f88a71","Type":"ContainerStarted","Data":"291217e041081f0a88e87908d58cdc2d8f224e0292c647f0bc553d9ca9d6b7e4"}
Jul 24 14:03:32.205569 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.205399    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:32.207009 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.206984    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" event={"ID":"194acd13-ab77-49eb-b176-071814be2b95","Type":"ContainerStarted","Data":"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"}
Jul 24 14:03:32.207170 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.207026    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" event={"ID":"194acd13-ab77-49eb-b176-071814be2b95","Type":"ContainerStarted","Data":"fa9dbc42e081eddafe8e577831f59d4c2722c5d928a182ba6e28bee6c8fd7822"}
Jul 24 14:03:32.207170 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.207072    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:32.209844 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.209816    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" event={"ID":"f617d5ec-8c0d-427e-9af6-37543d59cf22","Type":"ContainerStarted","Data":"ddf3a101adbd92544455d2a44793c4b18f05e2f33d80be89391690d2e0994968"}
Jul 24 14:03:32.210683 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.210664    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:32.226648 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.226589    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" podStartSLOduration=1.2265710890000001 podStartE2EDuration="1.226571089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:03:32.223405515 +0000 UTC m=+1776.265453401" watchObservedRunningTime="2026-07-24 14:03:32.226571089 +0000 UTC m=+1776.268618967"
Jul 24 14:03:32.250180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.250122    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" podStartSLOduration=1.7718315850000002 podStartE2EDuration="2.250103098s" totalImagesPullingTime="478.271513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:03:32.248839943 +0000 UTC m=+1776.290887828" watchObservedRunningTime="2026-07-24 14:03:32.250103098 +0000 UTC m=+1776.292150984"
Jul 24 14:03:32.267464 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:32.267380    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" podStartSLOduration=1.2673623059999999 podStartE2EDuration="1.267362306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:03:32.266327072 +0000 UTC m=+1776.308374967" watchObservedRunningTime="2026-07-24 14:03:32.267362306 +0000 UTC m=+1776.309410191"
Jul 24 14:03:39.229368 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:39.229337    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:03:43.219423 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:43.219380    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.56:9004: connect: connection refused"
Jul 24 14:03:43.219917 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:43.219883    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:03:53.219157 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:53.219079    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:03:56.501768 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:56.501741    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:03:56.502425 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:03:56.502408    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:04:06.265270 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.265227    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:04:06.265784 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.265498    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" podUID="f617d5ec-8c0d-427e-9af6-37543d59cf22" containerName="maas-api" containerID="cri-o://ddf3a101adbd92544455d2a44793c4b18f05e2f33d80be89391690d2e0994968" gracePeriod=30
Jul 24 14:04:06.344812 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.344762    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:04:06.345054 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.345021    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" podUID="194acd13-ab77-49eb-b176-071814be2b95" containerName="payload-processing" containerID="cri-o://7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e" gracePeriod=30
Jul 24 14:04:06.351495 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.351463    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:04:06.351707 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.351682    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerName="payload-pre-processing" containerID="cri-o://73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7" gracePeriod=30
Jul 24 14:04:06.391448 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.391408    2527 generic.go:356] "Generic (PLEG): container finished" podID="f617d5ec-8c0d-427e-9af6-37543d59cf22" containerID="ddf3a101adbd92544455d2a44793c4b18f05e2f33d80be89391690d2e0994968" exitCode=0
Jul 24 14:04:06.391598 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.391470    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" event={"ID":"f617d5ec-8c0d-427e-9af6-37543d59cf22","Type":"ContainerDied","Data":"ddf3a101adbd92544455d2a44793c4b18f05e2f33d80be89391690d2e0994968"}
Jul 24 14:04:06.643310 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.643283    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:04:06.646892 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.646867    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:04:06.650508 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.650482    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:04:06.742095 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742061    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls\") pod \"f617d5ec-8c0d-427e-9af6-37543d59cf22\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") "
Jul 24 14:04:06.742269 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742163    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8kct\" (UniqueName: \"kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct\") pod \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") "
Jul 24 14:04:06.742317 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742266    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume\") pod \"194acd13-ab77-49eb-b176-071814be2b95\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") "
Jul 24 14:04:06.742317 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742305    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume\") pod \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\" (UID: \"9b519a71-ac2d-4849-a94c-fd3d48f88a71\") "
Jul 24 14:04:06.742422 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742338    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca\") pod \"f617d5ec-8c0d-427e-9af6-37543d59cf22\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") "
Jul 24 14:04:06.742422 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742376    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9zbz\" (UniqueName: \"kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz\") pod \"f617d5ec-8c0d-427e-9af6-37543d59cf22\" (UID: \"f617d5ec-8c0d-427e-9af6-37543d59cf22\") "
Jul 24 14:04:06.742564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742427    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2x2w\" (UniqueName: \"kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w\") pod \"194acd13-ab77-49eb-b176-071814be2b95\" (UID: \"194acd13-ab77-49eb-b176-071814be2b95\") "
Jul 24 14:04:06.742748 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742716    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b519a71-ac2d-4849-a94c-fd3d48f88a71" (UID: "9b519a71-ac2d-4849-a94c-fd3d48f88a71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:04:06.742825 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.742721    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "194acd13-ab77-49eb-b176-071814be2b95" (UID: "194acd13-ab77-49eb-b176-071814be2b95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:04:06.743320 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.743289    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f617d5ec-8c0d-427e-9af6-37543d59cf22" (UID: "f617d5ec-8c0d-427e-9af6-37543d59cf22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:04:06.744718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.744694    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f617d5ec-8c0d-427e-9af6-37543d59cf22" (UID: "f617d5ec-8c0d-427e-9af6-37543d59cf22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:04:06.744819 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.744696    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct" (OuterVolumeSpecName: "kube-api-access-t8kct") pod "9b519a71-ac2d-4849-a94c-fd3d48f88a71" (UID: "9b519a71-ac2d-4849-a94c-fd3d48f88a71"). InnerVolumeSpecName "kube-api-access-t8kct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:04:06.744860 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.744810    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz" (OuterVolumeSpecName: "kube-api-access-m9zbz") pod "f617d5ec-8c0d-427e-9af6-37543d59cf22" (UID: "f617d5ec-8c0d-427e-9af6-37543d59cf22"). InnerVolumeSpecName "kube-api-access-m9zbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:04:06.745221 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.745196    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w" (OuterVolumeSpecName: "kube-api-access-q2x2w") pod "194acd13-ab77-49eb-b176-071814be2b95" (UID: "194acd13-ab77-49eb-b176-071814be2b95"). InnerVolumeSpecName "kube-api-access-q2x2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:04:06.844184 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844147    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kct\" (UniqueName: \"kubernetes.io/projected/9b519a71-ac2d-4849-a94c-fd3d48f88a71-kube-api-access-t8kct\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844184 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844180    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194acd13-ab77-49eb-b176-071814be2b95-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844184 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844189    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b519a71-ac2d-4849-a94c-fd3d48f88a71-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844200    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617d5ec-8c0d-427e-9af6-37543d59cf22-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844209    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zbz\" (UniqueName: \"kubernetes.io/projected/f617d5ec-8c0d-427e-9af6-37543d59cf22-kube-api-access-m9zbz\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844217    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2x2w\" (UniqueName: \"kubernetes.io/projected/194acd13-ab77-49eb-b176-071814be2b95-kube-api-access-q2x2w\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:06.844476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:06.844226    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617d5ec-8c0d-427e-9af6-37543d59cf22-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:04:07.398292 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.398258    2527 generic.go:356] "Generic (PLEG): container finished" podID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerID="73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7" exitCode=0
Jul 24 14:04:07.398734 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.398309    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"
Jul 24 14:04:07.398734 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.398345    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" event={"ID":"9b519a71-ac2d-4849-a94c-fd3d48f88a71","Type":"ContainerDied","Data":"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"}
Jul 24 14:04:07.398734 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.398396    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm" event={"ID":"9b519a71-ac2d-4849-a94c-fd3d48f88a71","Type":"ContainerDied","Data":"291217e041081f0a88e87908d58cdc2d8f224e0292c647f0bc553d9ca9d6b7e4"}
Jul 24 14:04:07.398734 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.398417    2527 scope.go:117] "RemoveContainer" containerID="73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"
Jul 24 14:04:07.399798 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.399777    2527 generic.go:356] "Generic (PLEG): container finished" podID="194acd13-ab77-49eb-b176-071814be2b95" containerID="7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e" exitCode=0
Jul 24 14:04:07.399857 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.399832    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"
Jul 24 14:04:07.399910 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.399873    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" event={"ID":"194acd13-ab77-49eb-b176-071814be2b95","Type":"ContainerDied","Data":"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"}
Jul 24 14:04:07.399950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.399919    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz" event={"ID":"194acd13-ab77-49eb-b176-071814be2b95","Type":"ContainerDied","Data":"fa9dbc42e081eddafe8e577831f59d4c2722c5d928a182ba6e28bee6c8fd7822"}
Jul 24 14:04:07.401718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.401698    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"
Jul 24 14:04:07.401718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.401707    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq" event={"ID":"f617d5ec-8c0d-427e-9af6-37543d59cf22","Type":"ContainerDied","Data":"6bec46487715c23b560fa4c6da4972c70e9503b70d520d14cddbf4000a2afb32"}
Jul 24 14:04:07.407885 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.407864    2527 scope.go:117] "RemoveContainer" containerID="73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"
Jul 24 14:04:07.408282 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:04:07.408250    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7\": container with ID starting with 73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7 not found: ID does not exist" containerID="73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"
Jul 24 14:04:07.408366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.408293    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7"} err="failed to get container status \"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7\": rpc error: code = NotFound desc = could not find container \"73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7\": container with ID starting with 73f03a97c740ca0d94afd2a1a6dede1c1c63500152bc17521f68ae39b81ae6b7 not found: ID does not exist"
Jul 24 14:04:07.408366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.408318    2527 scope.go:117] "RemoveContainer" containerID="7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"
Jul 24 14:04:07.418200 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.418182    2527 scope.go:117] "RemoveContainer" containerID="7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"
Jul 24 14:04:07.418531 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:04:07.418508    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e\": container with ID starting with 7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e not found: ID does not exist" containerID="7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"
Jul 24 14:04:07.418609 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.418542    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e"} err="failed to get container status \"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e\": rpc error: code = NotFound desc = could not find container \"7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e\": container with ID starting with 7e6acca9b7e613dd960438fee86cbd84d691b0a7820b8165e48e68f57d42171e not found: ID does not exist"
Jul 24 14:04:07.418609 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.418566    2527 scope.go:117] "RemoveContainer" containerID="ddf3a101adbd92544455d2a44793c4b18f05e2f33d80be89391690d2e0994968"
Jul 24 14:04:07.427449 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.427297    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:04:07.432560 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.432532    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9846c499-cb996bbd-26jkm"]
Jul 24 14:04:07.442870 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.442840    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:04:07.447790 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.447761    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9846c499-5cdd8bcc9d-dwpqq"]
Jul 24 14:04:07.461861 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.461836    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:04:07.465889 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:07.465863    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9846c499-6b6cc78cb6-zfxcz"]
Jul 24 14:04:08.514722 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:08.514692    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194acd13-ab77-49eb-b176-071814be2b95" path="/var/lib/kubelet/pods/194acd13-ab77-49eb-b176-071814be2b95/volumes"
Jul 24 14:04:08.515078 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:08.515040    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" path="/var/lib/kubelet/pods/9b519a71-ac2d-4849-a94c-fd3d48f88a71/volumes"
Jul 24 14:04:08.515375 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:04:08.515362    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f617d5ec-8c0d-427e-9af6-37543d59cf22" path="/var/lib/kubelet/pods/f617d5ec-8c0d-427e-9af6-37543d59cf22/volumes"
Jul 24 14:06:05.851893 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.851857    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:05.852461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852410    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerName="payload-pre-processing"
Jul 24 14:06:05.852461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852449    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerName="payload-pre-processing"
Jul 24 14:06:05.852608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852477    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194acd13-ab77-49eb-b176-071814be2b95" containerName="payload-processing"
Jul 24 14:06:05.852608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852485    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="194acd13-ab77-49eb-b176-071814be2b95" containerName="payload-processing"
Jul 24 14:06:05.852608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852516    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f617d5ec-8c0d-427e-9af6-37543d59cf22" containerName="maas-api"
Jul 24 14:06:05.852608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852525    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="f617d5ec-8c0d-427e-9af6-37543d59cf22" containerName="maas-api"
Jul 24 14:06:05.852810 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852631    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b519a71-ac2d-4849-a94c-fd3d48f88a71" containerName="payload-pre-processing"
Jul 24 14:06:05.852810 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852649    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="f617d5ec-8c0d-427e-9af6-37543d59cf22" containerName="maas-api"
Jul 24 14:06:05.852810 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.852659    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="194acd13-ab77-49eb-b176-071814be2b95" containerName="payload-processing"
Jul 24 14:06:05.855925 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.855901    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:05.858750 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.858725    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-296b2828-serving-cert\""
Jul 24 14:06:05.872060 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:05.872028    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:06.010496 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.010458    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfpj\" (UniqueName: \"kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.010686 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.010544    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.010686 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.010640    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.111699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.111604    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.111699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.111693    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.111903 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.111732    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfpj\" (UniqueName: \"kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.111903 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:06:06.111855    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-serving-cert: secret "maas-api-e2e-mt-296b2828-serving-cert" not found
Jul 24 14:06:06.111994 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:06:06.111944    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls podName:31af4997-266d-4a4a-9bc2-d97ae0571ea3 nodeName:}" failed. No retries permitted until 2026-07-24 14:06:06.611922387 +0000 UTC m=+1930.653970256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls") pod "maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" (UID: "31af4997-266d-4a4a-9bc2-d97ae0571ea3") : secret "maas-api-e2e-mt-296b2828-serving-cert" not found
Jul 24 14:06:06.112258 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.112238    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.121181 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.121148    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfpj\" (UniqueName: \"kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.616493 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.616414    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.619012 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.618987    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") pod \"maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.770794 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.770753    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:06.929132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.929092    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:06.932878 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:06:06.932850    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af4997_266d_4a4a_9bc2_d97ae0571ea3.slice/crio-14cd1296594598aac0955fb47ec38451679934806c4f42afc2d49d53e2f43fd2 WatchSource:0}: Error finding container 14cd1296594598aac0955fb47ec38451679934806c4f42afc2d49d53e2f43fd2: Status 404 returned error can't find the container with id 14cd1296594598aac0955fb47ec38451679934806c4f42afc2d49d53e2f43fd2
Jul 24 14:06:06.939311 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.939289    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:06.944207 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.944186    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:06.946290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.946272    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-296b2828\""
Jul 24 14:06:06.946386 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.946272    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-296b2828-dockercfg-9x6z4\""
Jul 24 14:06:06.951504 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.951466    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:06.981625 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.981592    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:06.985294 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.985266    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" event={"ID":"31af4997-266d-4a4a-9bc2-d97ae0571ea3","Type":"ContainerStarted","Data":"14cd1296594598aac0955fb47ec38451679934806c4f42afc2d49d53e2f43fd2"}
Jul 24 14:06:06.985462 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.985356    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:06.996748 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:06.996722    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:07.121479 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.121344    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.121479 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.121386    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2sm\" (UniqueName: \"kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.121479 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.121448    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.121717 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.121548    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69wr\" (UniqueName: \"kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.222718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.222676    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.222718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.222716    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2sm\" (UniqueName: \"kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.222972 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.222761    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.222972 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.222829    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69wr\" (UniqueName: \"kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.223596 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.223571    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.223707 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.223613    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.231596 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.231564    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69wr\" (UniqueName: \"kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr\") pod \"payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.232980 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.232954    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2sm\" (UniqueName: \"kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm\") pod \"payload-processing-e2e-mt-296b2828-5df587c654-99l49\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") " pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.256816 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.256781    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:07.296629 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.296591    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.419790 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.419763    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:07.423881 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:06:07.423839    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dd294e_5638_45c0_a5e6_6e250010ef3e.slice/crio-3052f46a1b58a01a1785ba5ff78340a2cb58864ec6506bb67e94dad33d2209aa WatchSource:0}: Error finding container 3052f46a1b58a01a1785ba5ff78340a2cb58864ec6506bb67e94dad33d2209aa: Status 404 returned error can't find the container with id 3052f46a1b58a01a1785ba5ff78340a2cb58864ec6506bb67e94dad33d2209aa
Jul 24 14:06:07.465191 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.465167    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:07.467811 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:06:07.467780    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee6756_ce20_49cb_b94c_33c36b84eb33.slice/crio-634ecb319d28f00745336467f64d7765c2620a2ecac41fc65e231abd6f26d9ad WatchSource:0}: Error finding container 634ecb319d28f00745336467f64d7765c2620a2ecac41fc65e231abd6f26d9ad: Status 404 returned error can't find the container with id 634ecb319d28f00745336467f64d7765c2620a2ecac41fc65e231abd6f26d9ad
Jul 24 14:06:07.989774 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.989674    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" event={"ID":"31af4997-266d-4a4a-9bc2-d97ae0571ea3","Type":"ContainerStarted","Data":"30cca626ad7a201440ea10ec2909fb08a4822ca16f1b5d36408cc6435bcf65aa"}
Jul 24 14:06:07.990201 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.989873    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:07.991304 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.991274    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" event={"ID":"59ee6756-ce20-49cb-b94c-33c36b84eb33","Type":"ContainerStarted","Data":"34d2d62aaec2618879c42438ed69a33fc9b5c3cfc306daed6868a511e07ff401"}
Jul 24 14:06:07.991460 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.991321    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" event={"ID":"59ee6756-ce20-49cb-b94c-33c36b84eb33","Type":"ContainerStarted","Data":"634ecb319d28f00745336467f64d7765c2620a2ecac41fc65e231abd6f26d9ad"}
Jul 24 14:06:07.991460 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.991385    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:07.992909 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.992885    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" event={"ID":"73dd294e-5638-45c0-a5e6-6e250010ef3e","Type":"ContainerStarted","Data":"36222784ee630a50b2a7ba05e2f15bc8e81f1740880bdbb0a5ddc288c813f614"}
Jul 24 14:06:07.993028 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.992927    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" event={"ID":"73dd294e-5638-45c0-a5e6-6e250010ef3e","Type":"ContainerStarted","Data":"3052f46a1b58a01a1785ba5ff78340a2cb58864ec6506bb67e94dad33d2209aa"}
Jul 24 14:06:07.993028 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:07.993023    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:08.015178 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:08.015116    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" podStartSLOduration=2.242498121 podStartE2EDuration="3.015094035s" totalImagesPullingTime="772.595914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:08.01274319 +0000 UTC m=+1932.054791056" watchObservedRunningTime="2026-07-24 14:06:08.015094035 +0000 UTC m=+1932.057141920"
Jul 24 14:06:08.035467 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:08.035396    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" podStartSLOduration=2.035377214 podStartE2EDuration="2.035377214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:08.033828615 +0000 UTC m=+1932.075876501" watchObservedRunningTime="2026-07-24 14:06:08.035377214 +0000 UTC m=+1932.077425077"
Jul 24 14:06:08.058523 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:08.058461    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" podStartSLOduration=2.058444734 podStartE2EDuration="2.058444734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:08.056997076 +0000 UTC m=+1932.099044962" watchObservedRunningTime="2026-07-24 14:06:08.058444734 +0000 UTC m=+1932.100492610"
Jul 24 14:06:14.004791 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:14.004757    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:19.001205 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:19.001171    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:19.001849 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:19.001282    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:28.819185 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.819151    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:28.821752 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.819364    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerName="maas-api" containerID="cri-o://30cca626ad7a201440ea10ec2909fb08a4822ca16f1b5d36408cc6435bcf65aa" gracePeriod=30
Jul 24 14:06:28.899345 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.899307    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:28.899617 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.899590    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerName="payload-processing" containerID="cri-o://34d2d62aaec2618879c42438ed69a33fc9b5c3cfc306daed6868a511e07ff401" gracePeriod=30
Jul 24 14:06:28.914339 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.914305    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:28.914591 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.914566    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerName="payload-pre-processing" containerID="cri-o://36222784ee630a50b2a7ba05e2f15bc8e81f1740880bdbb0a5ddc288c813f614" gracePeriod=30
Jul 24 14:06:28.999950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.999909    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.59:9004: connect: connection refused"
Jul 24 14:06:29.000124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.999910    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.60:9004: connect: connection refused"
Jul 24 14:06:29.000124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:28.999916    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": dial tcp 10.132.0.58:8443: connect: connection refused"
Jul 24 14:06:29.099804 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.099771    2527 generic.go:356] "Generic (PLEG): container finished" podID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerID="34d2d62aaec2618879c42438ed69a33fc9b5c3cfc306daed6868a511e07ff401" exitCode=0
Jul 24 14:06:29.100048 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.099881    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" event={"ID":"59ee6756-ce20-49cb-b94c-33c36b84eb33","Type":"ContainerDied","Data":"34d2d62aaec2618879c42438ed69a33fc9b5c3cfc306daed6868a511e07ff401"}
Jul 24 14:06:29.101480 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.101425    2527 generic.go:356] "Generic (PLEG): container finished" podID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerID="36222784ee630a50b2a7ba05e2f15bc8e81f1740880bdbb0a5ddc288c813f614" exitCode=0
Jul 24 14:06:29.101624 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.101482    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" event={"ID":"73dd294e-5638-45c0-a5e6-6e250010ef3e","Type":"ContainerDied","Data":"36222784ee630a50b2a7ba05e2f15bc8e81f1740880bdbb0a5ddc288c813f614"}
Jul 24 14:06:29.103327 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.103301    2527 generic.go:356] "Generic (PLEG): container finished" podID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerID="30cca626ad7a201440ea10ec2909fb08a4822ca16f1b5d36408cc6435bcf65aa" exitCode=0
Jul 24 14:06:29.103456 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.103354    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" event={"ID":"31af4997-266d-4a4a-9bc2-d97ae0571ea3","Type":"ContainerDied","Data":"30cca626ad7a201440ea10ec2909fb08a4822ca16f1b5d36408cc6435bcf65aa"}
Jul 24 14:06:29.174871 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.174838    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:29.206902 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.206872    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:29.216798 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.216771    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:29.220584 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.220560    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume\") pod \"59ee6756-ce20-49cb-b94c-33c36b84eb33\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") "
Jul 24 14:06:29.220712 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.220600    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw2sm\" (UniqueName: \"kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm\") pod \"59ee6756-ce20-49cb-b94c-33c36b84eb33\" (UID: \"59ee6756-ce20-49cb-b94c-33c36b84eb33\") "
Jul 24 14:06:29.220964 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.220940    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59ee6756-ce20-49cb-b94c-33c36b84eb33" (UID: "59ee6756-ce20-49cb-b94c-33c36b84eb33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:06:29.222849 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.222819    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm" (OuterVolumeSpecName: "kube-api-access-hw2sm") pod "59ee6756-ce20-49cb-b94c-33c36b84eb33" (UID: "59ee6756-ce20-49cb-b94c-33c36b84eb33"). InnerVolumeSpecName "kube-api-access-hw2sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.321895    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") pod \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") "
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322011    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z69wr\" (UniqueName: \"kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr\") pod \"73dd294e-5638-45c0-a5e6-6e250010ef3e\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") "
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322067    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzfpj\" (UniqueName: \"kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj\") pod \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") "
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322126    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca\") pod \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\" (UID: \"31af4997-266d-4a4a-9bc2-d97ae0571ea3\") "
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322158    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume\") pod \"73dd294e-5638-45c0-a5e6-6e250010ef3e\" (UID: \"73dd294e-5638-45c0-a5e6-6e250010ef3e\") "
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322673    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ee6756-ce20-49cb-b94c-33c36b84eb33-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.322694    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2sm\" (UniqueName: \"kubernetes.io/projected/59ee6756-ce20-49cb-b94c-33c36b84eb33-kube-api-access-hw2sm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.325550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.323133    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73dd294e-5638-45c0-a5e6-6e250010ef3e" (UID: "73dd294e-5638-45c0-a5e6-6e250010ef3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:06:29.328621 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.328569    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31af4997-266d-4a4a-9bc2-d97ae0571ea3" (UID: "31af4997-266d-4a4a-9bc2-d97ae0571ea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:06:29.329151 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.329126    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31af4997-266d-4a4a-9bc2-d97ae0571ea3" (UID: "31af4997-266d-4a4a-9bc2-d97ae0571ea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:06:29.329260 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.329222    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr" (OuterVolumeSpecName: "kube-api-access-z69wr") pod "73dd294e-5638-45c0-a5e6-6e250010ef3e" (UID: "73dd294e-5638-45c0-a5e6-6e250010ef3e"). InnerVolumeSpecName "kube-api-access-z69wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:06:29.330356 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.330336    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj" (OuterVolumeSpecName: "kube-api-access-hzfpj") pod "31af4997-266d-4a4a-9bc2-d97ae0571ea3" (UID: "31af4997-266d-4a4a-9bc2-d97ae0571ea3"). InnerVolumeSpecName "kube-api-access-hzfpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:06:29.423824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.423740    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31af4997-266d-4a4a-9bc2-d97ae0571ea3-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.423824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.423770    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z69wr\" (UniqueName: \"kubernetes.io/projected/73dd294e-5638-45c0-a5e6-6e250010ef3e-kube-api-access-z69wr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.423824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.423781    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfpj\" (UniqueName: \"kubernetes.io/projected/31af4997-266d-4a4a-9bc2-d97ae0571ea3-kube-api-access-hzfpj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.423824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.423790    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31af4997-266d-4a4a-9bc2-d97ae0571ea3-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:29.423824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:29.423798    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73dd294e-5638-45c0-a5e6-6e250010ef3e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:06:30.109574 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.109488    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"
Jul 24 14:06:30.110022 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.109482    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b" event={"ID":"31af4997-266d-4a4a-9bc2-d97ae0571ea3","Type":"ContainerDied","Data":"14cd1296594598aac0955fb47ec38451679934806c4f42afc2d49d53e2f43fd2"}
Jul 24 14:06:30.110022 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.109643    2527 scope.go:117] "RemoveContainer" containerID="30cca626ad7a201440ea10ec2909fb08a4822ca16f1b5d36408cc6435bcf65aa"
Jul 24 14:06:30.110978 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.110902    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"
Jul 24 14:06:30.110978 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.110919    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49" event={"ID":"59ee6756-ce20-49cb-b94c-33c36b84eb33","Type":"ContainerDied","Data":"634ecb319d28f00745336467f64d7765c2620a2ecac41fc65e231abd6f26d9ad"}
Jul 24 14:06:30.112586 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.112556    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2" event={"ID":"73dd294e-5638-45c0-a5e6-6e250010ef3e","Type":"ContainerDied","Data":"3052f46a1b58a01a1785ba5ff78340a2cb58864ec6506bb67e94dad33d2209aa"}
Jul 24 14:06:30.112739 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.112597    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"
Jul 24 14:06:30.120530 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.120512    2527 scope.go:117] "RemoveContainer" containerID="34d2d62aaec2618879c42438ed69a33fc9b5c3cfc306daed6868a511e07ff401"
Jul 24 14:06:30.130624 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.130603    2527 scope.go:117] "RemoveContainer" containerID="36222784ee630a50b2a7ba05e2f15bc8e81f1740880bdbb0a5ddc288c813f614"
Jul 24 14:06:30.139057 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.138948    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:30.147770 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.147746    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-296b2828-7869f5bf4c-22t9b"]
Jul 24 14:06:30.160881 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.160854    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:30.168709 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.168686    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-296b2828-86c69cb45b-7twx2"]
Jul 24 14:06:30.181091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.181063    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:30.184741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.184715    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-296b2828-5df587c654-99l49"]
Jul 24 14:06:30.513578 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.513501    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" path="/var/lib/kubelet/pods/31af4997-266d-4a4a-9bc2-d97ae0571ea3/volumes"
Jul 24 14:06:30.513884 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.513871    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" path="/var/lib/kubelet/pods/59ee6756-ce20-49cb-b94c-33c36b84eb33/volumes"
Jul 24 14:06:30.514199 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:06:30.514187    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" path="/var/lib/kubelet/pods/73dd294e-5638-45c0-a5e6-6e250010ef3e/volumes"
Jul 24 14:07:05.204821 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.204788    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205207    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerName="payload-processing"
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205221    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerName="payload-processing"
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205245    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerName="payload-pre-processing"
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205251    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerName="payload-pre-processing"
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205257    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerName="maas-api"
Jul 24 14:07:05.205303 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205262    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerName="maas-api"
Jul 24 14:07:05.205641 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205325    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="31af4997-266d-4a4a-9bc2-d97ae0571ea3" containerName="maas-api"
Jul 24 14:07:05.205641 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205332    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ee6756-ce20-49cb-b94c-33c36b84eb33" containerName="payload-processing"
Jul 24 14:07:05.205641 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.205338    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="73dd294e-5638-45c0-a5e6-6e250010ef3e" containerName="payload-pre-processing"
Jul 24 14:07:05.208454 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.208421    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.211801 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.211781    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3098bd50-serving-cert\""
Jul 24 14:07:05.237356 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.237324    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:07:05.253067 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.253034    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.253252 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.253083    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.253252 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.253171    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4vq\" (UniqueName: \"kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.354059 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.354023    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4vq\" (UniqueName: \"kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.354285 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.354128    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.354285 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.354161    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.354285 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:07:05.354274    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-serving-cert: secret "maas-api-e2e-mt-3098bd50-serving-cert" not found
Jul 24 14:07:05.354498 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:07:05.354352    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls podName:4e494abb-cc2d-4662-9433-5f081f95a4b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:07:05.854331034 +0000 UTC m=+1989.896378897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls") pod "maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" (UID: "4e494abb-cc2d-4662-9433-5f081f95a4b8") : secret "maas-api-e2e-mt-3098bd50-serving-cert" not found
Jul 24 14:07:05.354980 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.354961    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.370007 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.369974    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4vq\" (UniqueName: \"kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.858690 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.858654    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:05.861478 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:05.861429    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") pod \"maas-api-e2e-mt-3098bd50-79657b547f-lnpf4\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:06.120377 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.120269    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:06.289306 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.289275    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:07:06.292138 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:07:06.292107    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e494abb_cc2d_4662_9433_5f081f95a4b8.slice/crio-3262a9f18a683f488e60a36373de3fb2ec94d6df519885a9ae70c0e00050591e WatchSource:0}: Error finding container 3262a9f18a683f488e60a36373de3fb2ec94d6df519885a9ae70c0e00050591e: Status 404 returned error can't find the container with id 3262a9f18a683f488e60a36373de3fb2ec94d6df519885a9ae70c0e00050591e
Jul 24 14:07:06.293612 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.293592    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:07:06.298768 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.298729    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" event={"ID":"4e494abb-cc2d-4662-9433-5f081f95a4b8","Type":"ContainerStarted","Data":"3262a9f18a683f488e60a36373de3fb2ec94d6df519885a9ae70c0e00050591e"}
Jul 24 14:07:06.343640 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.343606    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:07:06.348323 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.348302    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.350711 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.350684    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3098bd50\""
Jul 24 14:07:06.350824 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.350734    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3098bd50-dockercfg-9plf5\""
Jul 24 14:07:06.359090 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.359057    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:07:06.446996 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.446888    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:07:06.450729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.450706    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.453220 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:07:06.453178    2527 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.470121 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.470088    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv2d\" (UniqueName: \"kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.470267 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.470155    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.483397 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.483366    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:07:06.570924 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.570886    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94wp\" (UniqueName: \"kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.571168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.570937    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv2d\" (UniqueName: \"kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.571168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.571062    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.571300 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.571210    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.571822 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.571797    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.585961 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.585929    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv2d\" (UniqueName: \"kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d\") pod \"payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.660257 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.660216    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:06.672302 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.672205    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.672372 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.672311    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94wp\" (UniqueName: \"kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.673109 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.673080    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.684106 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.684052    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94wp\" (UniqueName: \"kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp\") pod \"payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") " pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.761823 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.761775    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:06.821624 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.821592    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:07:06.823861 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:07:06.823818    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde016d47_644b_4307_86a4_dba77b86e321.slice/crio-3d4b2fb2fd8ae7cff533cd6046041de33c8e6b24c06489954d7f59aa08a75283 WatchSource:0}: Error finding container 3d4b2fb2fd8ae7cff533cd6046041de33c8e6b24c06489954d7f59aa08a75283: Status 404 returned error can't find the container with id 3d4b2fb2fd8ae7cff533cd6046041de33c8e6b24c06489954d7f59aa08a75283
Jul 24 14:07:06.939212 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:06.939084    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:07:06.941450 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:07:06.941399    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ee9c77_60cf_4dc2_9252_619ca00c5217.slice/crio-6097b0adae1d7cc89093b16cd62f3e525b7e02a74bb72c6e5ad851af55bb3a95 WatchSource:0}: Error finding container 6097b0adae1d7cc89093b16cd62f3e525b7e02a74bb72c6e5ad851af55bb3a95: Status 404 returned error can't find the container with id 6097b0adae1d7cc89093b16cd62f3e525b7e02a74bb72c6e5ad851af55bb3a95
Jul 24 14:07:07.306007 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.305913    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" event={"ID":"4e494abb-cc2d-4662-9433-5f081f95a4b8","Type":"ContainerStarted","Data":"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"}
Jul 24 14:07:07.306007 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.305988    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:07.307599 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.307569    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" event={"ID":"73ee9c77-60cf-4dc2-9252-619ca00c5217","Type":"ContainerStarted","Data":"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"}
Jul 24 14:07:07.307741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.307619    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" event={"ID":"73ee9c77-60cf-4dc2-9252-619ca00c5217","Type":"ContainerStarted","Data":"6097b0adae1d7cc89093b16cd62f3e525b7e02a74bb72c6e5ad851af55bb3a95"}
Jul 24 14:07:07.307741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.307647    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:07:07.309179 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.309152    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" event={"ID":"de016d47-644b-4307-86a4-dba77b86e321","Type":"ContainerStarted","Data":"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"}
Jul 24 14:07:07.309315 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.309198    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" event={"ID":"de016d47-644b-4307-86a4-dba77b86e321","Type":"ContainerStarted","Data":"3d4b2fb2fd8ae7cff533cd6046041de33c8e6b24c06489954d7f59aa08a75283"}
Jul 24 14:07:07.309315 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.309228    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:07.328995 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.328934    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" podStartSLOduration=1.770346492 podStartE2EDuration="2.328914634s" totalImagesPullingTime="558.568142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:07.328418665 +0000 UTC m=+1991.370466550" watchObservedRunningTime="2026-07-24 14:07:07.328914634 +0000 UTC m=+1991.370962519"
Jul 24 14:07:07.355412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.355360    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" podStartSLOduration=1.355343268 podStartE2EDuration="1.355343268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:07.354706632 +0000 UTC m=+1991.396754517" watchObservedRunningTime="2026-07-24 14:07:07.355343268 +0000 UTC m=+1991.397391152"
Jul 24 14:07:07.381930 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:07.381878    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" podStartSLOduration=1.381861293 podStartE2EDuration="1.381861293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:07.378718734 +0000 UTC m=+1991.420766620" watchObservedRunningTime="2026-07-24 14:07:07.381861293 +0000 UTC m=+1991.423909177"
Jul 24 14:07:13.319716 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:13.319689    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:07:18.316379 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:18.316343    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:07:18.316806 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:07:18.316425    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:08:20.765762 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.765671    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:08:20.766316 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.765870    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" podUID="4e494abb-cc2d-4662-9433-5f081f95a4b8" containerName="maas-api" containerID="cri-o://bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900" gracePeriod=30
Jul 24 14:08:20.884811 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.884774    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:08:20.885083 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.885034    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" containerName="payload-processing" containerID="cri-o://7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb" gracePeriod=30
Jul 24 14:08:20.908481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.908449    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:08:20.908718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:20.908694    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" podUID="de016d47-644b-4307-86a4-dba77b86e321" containerName="payload-pre-processing" containerID="cri-o://89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f" gracePeriod=30
Jul 24 14:08:21.034786 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.034759    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:08:21.148924 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.148890    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca\") pod \"4e494abb-cc2d-4662-9433-5f081f95a4b8\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") "
Jul 24 14:08:21.148924 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.148929    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") pod \"4e494abb-cc2d-4662-9433-5f081f95a4b8\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") "
Jul 24 14:08:21.149177 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.149090    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj4vq\" (UniqueName: \"kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq\") pod \"4e494abb-cc2d-4662-9433-5f081f95a4b8\" (UID: \"4e494abb-cc2d-4662-9433-5f081f95a4b8\") "
Jul 24 14:08:21.149325 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.149299    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e494abb-cc2d-4662-9433-5f081f95a4b8" (UID: "4e494abb-cc2d-4662-9433-5f081f95a4b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:08:21.149479 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.149463    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e494abb-cc2d-4662-9433-5f081f95a4b8-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.152486 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.152456    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e494abb-cc2d-4662-9433-5f081f95a4b8" (UID: "4e494abb-cc2d-4662-9433-5f081f95a4b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:08:21.152486 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.152466    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq" (OuterVolumeSpecName: "kube-api-access-wj4vq") pod "4e494abb-cc2d-4662-9433-5f081f95a4b8" (UID: "4e494abb-cc2d-4662-9433-5f081f95a4b8"). InnerVolumeSpecName "kube-api-access-wj4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:08:21.173757 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.173733    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:08:21.184726 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.184703    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:08:21.249955 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.249922    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e494abb-cc2d-4662-9433-5f081f95a4b8-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.249955 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.249955    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4vq\" (UniqueName: \"kubernetes.io/projected/4e494abb-cc2d-4662-9433-5f081f95a4b8-kube-api-access-wj4vq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.350621 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.350574    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume\") pod \"de016d47-644b-4307-86a4-dba77b86e321\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") "
Jul 24 14:08:21.350823 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.350669    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume\") pod \"73ee9c77-60cf-4dc2-9252-619ca00c5217\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") "
Jul 24 14:08:21.350823 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.350695    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j94wp\" (UniqueName: \"kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp\") pod \"73ee9c77-60cf-4dc2-9252-619ca00c5217\" (UID: \"73ee9c77-60cf-4dc2-9252-619ca00c5217\") "
Jul 24 14:08:21.350823 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.350753    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvv2d\" (UniqueName: \"kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d\") pod \"de016d47-644b-4307-86a4-dba77b86e321\" (UID: \"de016d47-644b-4307-86a4-dba77b86e321\") "
Jul 24 14:08:21.351026 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.350985    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de016d47-644b-4307-86a4-dba77b86e321" (UID: "de016d47-644b-4307-86a4-dba77b86e321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:08:21.351123 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.351103    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73ee9c77-60cf-4dc2-9252-619ca00c5217" (UID: "73ee9c77-60cf-4dc2-9252-619ca00c5217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:08:21.353068 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.353046    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d" (OuterVolumeSpecName: "kube-api-access-bvv2d") pod "de016d47-644b-4307-86a4-dba77b86e321" (UID: "de016d47-644b-4307-86a4-dba77b86e321"). InnerVolumeSpecName "kube-api-access-bvv2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:08:21.353150 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.353117    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp" (OuterVolumeSpecName: "kube-api-access-j94wp") pod "73ee9c77-60cf-4dc2-9252-619ca00c5217" (UID: "73ee9c77-60cf-4dc2-9252-619ca00c5217"). InnerVolumeSpecName "kube-api-access-j94wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:08:21.451473 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.451400    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ee9c77-60cf-4dc2-9252-619ca00c5217-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.451473 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.451461    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j94wp\" (UniqueName: \"kubernetes.io/projected/73ee9c77-60cf-4dc2-9252-619ca00c5217-kube-api-access-j94wp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.451473 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.451474    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvv2d\" (UniqueName: \"kubernetes.io/projected/de016d47-644b-4307-86a4-dba77b86e321-kube-api-access-bvv2d\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.451473 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.451482    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de016d47-644b-4307-86a4-dba77b86e321-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:08:21.678481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.678373    2527 generic.go:356] "Generic (PLEG): container finished" podID="73ee9c77-60cf-4dc2-9252-619ca00c5217" containerID="7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb" exitCode=0
Jul 24 14:08:21.678481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.678419    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"
Jul 24 14:08:21.678481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.678466    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" event={"ID":"73ee9c77-60cf-4dc2-9252-619ca00c5217","Type":"ContainerDied","Data":"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"}
Jul 24 14:08:21.678740 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.678526    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n" event={"ID":"73ee9c77-60cf-4dc2-9252-619ca00c5217","Type":"ContainerDied","Data":"6097b0adae1d7cc89093b16cd62f3e525b7e02a74bb72c6e5ad851af55bb3a95"}
Jul 24 14:08:21.678740 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.678555    2527 scope.go:117] "RemoveContainer" containerID="7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"
Jul 24 14:08:21.684080 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.684047    2527 generic.go:356] "Generic (PLEG): container finished" podID="de016d47-644b-4307-86a4-dba77b86e321" containerID="89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f" exitCode=0
Jul 24 14:08:21.684233 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.684114    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"
Jul 24 14:08:21.684233 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.684156    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" event={"ID":"de016d47-644b-4307-86a4-dba77b86e321","Type":"ContainerDied","Data":"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"}
Jul 24 14:08:21.684233 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.684225    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7" event={"ID":"de016d47-644b-4307-86a4-dba77b86e321","Type":"ContainerDied","Data":"3d4b2fb2fd8ae7cff533cd6046041de33c8e6b24c06489954d7f59aa08a75283"}
Jul 24 14:08:21.685940 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.685904    2527 generic.go:356] "Generic (PLEG): container finished" podID="4e494abb-cc2d-4662-9433-5f081f95a4b8" containerID="bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900" exitCode=0
Jul 24 14:08:21.686061 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.685954    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" event={"ID":"4e494abb-cc2d-4662-9433-5f081f95a4b8","Type":"ContainerDied","Data":"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"}
Jul 24 14:08:21.686061 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.685958    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"
Jul 24 14:08:21.686061 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.685996    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4" event={"ID":"4e494abb-cc2d-4662-9433-5f081f95a4b8","Type":"ContainerDied","Data":"3262a9f18a683f488e60a36373de3fb2ec94d6df519885a9ae70c0e00050591e"}
Jul 24 14:08:21.690879 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.690855    2527 scope.go:117] "RemoveContainer" containerID="7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"
Jul 24 14:08:21.691214 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:08:21.691193    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb\": container with ID starting with 7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb not found: ID does not exist" containerID="7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"
Jul 24 14:08:21.691272 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.691228    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb"} err="failed to get container status \"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb\": rpc error: code = NotFound desc = could not find container \"7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb\": container with ID starting with 7944f452863db318ee2b3472a14c7ae5ac8d924b9788de066adf605879213edb not found: ID does not exist"
Jul 24 14:08:21.691272 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.691265    2527 scope.go:117] "RemoveContainer" containerID="89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"
Jul 24 14:08:21.700696 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.700678    2527 scope.go:117] "RemoveContainer" containerID="89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"
Jul 24 14:08:21.700988 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:08:21.700969    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f\": container with ID starting with 89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f not found: ID does not exist" containerID="89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"
Jul 24 14:08:21.701044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.700996    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f"} err="failed to get container status \"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f\": rpc error: code = NotFound desc = could not find container \"89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f\": container with ID starting with 89b995450a24e84439ec72480940169979650372ab0bf1e215209852ef67869f not found: ID does not exist"
Jul 24 14:08:21.701044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.701015    2527 scope.go:117] "RemoveContainer" containerID="bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"
Jul 24 14:08:21.709690 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.709671    2527 scope.go:117] "RemoveContainer" containerID="bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"
Jul 24 14:08:21.709977 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:08:21.709958    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900\": container with ID starting with bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900 not found: ID does not exist" containerID="bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"
Jul 24 14:08:21.710024 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.709988    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900"} err="failed to get container status \"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900\": rpc error: code = NotFound desc = could not find container \"bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900\": container with ID starting with bb88c1a9b489a878b5f9d51edd795152f1b73bfac290438cf2236f04bd53b900 not found: ID does not exist"
Jul 24 14:08:21.720217 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.720184    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:08:21.728133 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.728100    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3098bd50-7dc7755978-6r97n"]
Jul 24 14:08:21.751450 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.751393    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:08:21.759845 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.759808    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3098bd50-79657b547f-lnpf4"]
Jul 24 14:08:21.780344 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.780302    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:08:21.793876 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:21.793847    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3098bd50-76b597f6dd-8phs7"]
Jul 24 14:08:22.514385 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:22.514347    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e494abb-cc2d-4662-9433-5f081f95a4b8" path="/var/lib/kubelet/pods/4e494abb-cc2d-4662-9433-5f081f95a4b8/volumes"
Jul 24 14:08:22.514773 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:22.514758    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" path="/var/lib/kubelet/pods/73ee9c77-60cf-4dc2-9252-619ca00c5217/volumes"
Jul 24 14:08:22.515077 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:22.515064    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de016d47-644b-4307-86a4-dba77b86e321" path="/var/lib/kubelet/pods/de016d47-644b-4307-86a4-dba77b86e321/volumes"
Jul 24 14:08:56.529398 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:56.529355    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:08:56.531334 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:08:56.531312    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:10:23.397280 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397199    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397644    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" containerName="payload-processing"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397659    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" containerName="payload-processing"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397677    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de016d47-644b-4307-86a4-dba77b86e321" containerName="payload-pre-processing"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397682    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="de016d47-644b-4307-86a4-dba77b86e321" containerName="payload-pre-processing"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397698    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e494abb-cc2d-4662-9433-5f081f95a4b8" containerName="maas-api"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397703    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e494abb-cc2d-4662-9433-5f081f95a4b8" containerName="maas-api"
Jul 24 14:10:23.397764 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397768    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ee9c77-60cf-4dc2-9252-619ca00c5217" containerName="payload-processing"
Jul 24 14:10:23.397994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397777    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e494abb-cc2d-4662-9433-5f081f95a4b8" containerName="maas-api"
Jul 24 14:10:23.397994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.397784    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="de016d47-644b-4307-86a4-dba77b86e321" containerName="payload-pre-processing"
Jul 24 14:10:23.401101 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.401081    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.406325 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.406299    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c21e7d-serving-cert\""
Jul 24 14:10:23.422130 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.422093    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:10:23.473881 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.473844    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.474074 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.473909    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpjwg\" (UniqueName: \"kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.474074 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.473978    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.575467 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.575392    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.575656 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.575534    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpjwg\" (UniqueName: \"kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.575656 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.575579    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.575775 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:10:23.575750    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-serving-cert: secret "maas-api-e2e-inf-a-c21e7d-serving-cert" not found
Jul 24 14:10:23.575855 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:10:23.575842    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls podName:fb09ed56-a69c-45bb-8977-d17dadc80570 nodeName:}" failed. No retries permitted until 2026-07-24 14:10:24.075821658 +0000 UTC m=+2188.117869528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls") pod "maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" (UID: "fb09ed56-a69c-45bb-8977-d17dadc80570") : secret "maas-api-e2e-inf-a-c21e7d-serving-cert" not found
Jul 24 14:10:23.576172 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.576153    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:23.590268 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:23.590229    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpjwg\" (UniqueName: \"kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:24.079974 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.079937    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:24.082670 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.082641    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:24.312165 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.312131    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:24.364043 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.363958    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:10:24.368361 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.368338    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.371537 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.371221    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c21e7d\""
Jul 24 14:10:24.371537 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.371256    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c21e7d-dockercfg-5fwpv\""
Jul 24 14:10:24.378754 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.378690    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:10:24.480520 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.480399    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:10:24.482462 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.482419    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:10:24.482845 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:24.482819    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb09ed56_a69c_45bb_8977_d17dadc80570.slice/crio-8d9740b24002f5a9924835f886386e9f05019ae8dccb78c1e882e2102d9fe463 WatchSource:0}: Error finding container 8d9740b24002f5a9924835f886386e9f05019ae8dccb78c1e882e2102d9fe463: Status 404 returned error can't find the container with id 8d9740b24002f5a9924835f886386e9f05019ae8dccb78c1e882e2102d9fe463
Jul 24 14:10:24.483181 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.483152    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.483236 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.483202    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zj9p\" (UniqueName: \"kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.486304 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.486286    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.496908 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.496874    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:10:24.584240 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.584203    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99ch\" (UniqueName: \"kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.584525 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.584246    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.584525 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.584322    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.584525 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.584348    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zj9p\" (UniqueName: \"kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.585132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.585110    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.602099 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.602061    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zj9p\" (UniqueName: \"kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p\") pod \"payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.685147 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.685062    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99ch\" (UniqueName: \"kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.685147 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.685100    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.685378 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.685225    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:24.685862 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.685844    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.697466 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.697414    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99ch\" (UniqueName: \"kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch\") pod \"payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.802935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.802904    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:24.838124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.838100    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:10:24.840122 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:24.840086    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7810a6_0f17_4fe8_b195_8c83b0dd87bf.slice/crio-8aab6ba67308c3df2676deea0b0193d930dec761c022b320d1634994b97cb9c8 WatchSource:0}: Error finding container 8aab6ba67308c3df2676deea0b0193d930dec761c022b320d1634994b97cb9c8: Status 404 returned error can't find the container with id 8aab6ba67308c3df2676deea0b0193d930dec761c022b320d1634994b97cb9c8
Jul 24 14:10:24.960306 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:24.960222    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:10:24.962648 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:24.962620    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61dddd9_cb45_4988_9621_3e8cc135579b.slice/crio-5a6be107b08072e992415955841ed90d277fb2b6345f833d75960b2a85eae4fa WatchSource:0}: Error finding container 5a6be107b08072e992415955841ed90d277fb2b6345f833d75960b2a85eae4fa: Status 404 returned error can't find the container with id 5a6be107b08072e992415955841ed90d277fb2b6345f833d75960b2a85eae4fa
Jul 24 14:10:25.291798 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.291592    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" event={"ID":"fb09ed56-a69c-45bb-8977-d17dadc80570","Type":"ContainerStarted","Data":"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"}
Jul 24 14:10:25.291798 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.291659    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" event={"ID":"fb09ed56-a69c-45bb-8977-d17dadc80570","Type":"ContainerStarted","Data":"8d9740b24002f5a9924835f886386e9f05019ae8dccb78c1e882e2102d9fe463"}
Jul 24 14:10:25.292052 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.291855    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:25.293943 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.293910    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" event={"ID":"d61dddd9-cb45-4988-9621-3e8cc135579b","Type":"ContainerStarted","Data":"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"}
Jul 24 14:10:25.294107 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.293967    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" event={"ID":"d61dddd9-cb45-4988-9621-3e8cc135579b","Type":"ContainerStarted","Data":"5a6be107b08072e992415955841ed90d277fb2b6345f833d75960b2a85eae4fa"}
Jul 24 14:10:25.294107 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.294014    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:25.295579 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.295554    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" event={"ID":"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf","Type":"ContainerStarted","Data":"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"}
Jul 24 14:10:25.295705 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.295598    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" event={"ID":"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf","Type":"ContainerStarted","Data":"8aab6ba67308c3df2676deea0b0193d930dec761c022b320d1634994b97cb9c8"}
Jul 24 14:10:25.295705 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.295627    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:25.350589 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.350391    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" podStartSLOduration=1.8747911369999999 podStartE2EDuration="2.35036995s" totalImagesPullingTime="475.578813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:25.349990272 +0000 UTC m=+2189.392038179" watchObservedRunningTime="2026-07-24 14:10:25.35036995 +0000 UTC m=+2189.392417835"
Jul 24 14:10:25.407119 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.407060    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" podStartSLOduration=1.407043911 podStartE2EDuration="1.407043911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:25.40335831 +0000 UTC m=+2189.445406195" watchObservedRunningTime="2026-07-24 14:10:25.407043911 +0000 UTC m=+2189.449091796"
Jul 24 14:10:25.437646 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:25.437581    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" podStartSLOduration=1.437559354 podStartE2EDuration="1.437559354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:25.436156749 +0000 UTC m=+2189.478204636" watchObservedRunningTime="2026-07-24 14:10:25.437559354 +0000 UTC m=+2189.479607246"
Jul 24 14:10:31.308493 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.308459    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:10:31.427737 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.427706    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:10:31.431227 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.431209    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.433383 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.433356    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d52a8f-serving-cert\""
Jul 24 14:10:31.456008 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.455965    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:10:31.554629 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.554589    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6v5\" (UniqueName: \"kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.554816 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.554643    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.554816 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.554733    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.655613 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.655567    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6v5\" (UniqueName: \"kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.655613 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.655618    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.655880 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:10:31.655714    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-serving-cert: secret "maas-api-e2e-inf-b-d52a8f-serving-cert" not found
Jul 24 14:10:31.655880 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:10:31.655770    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls podName:883c3ad8-259f-436f-b48a-a825103617ad nodeName:}" failed. No retries permitted until 2026-07-24 14:10:32.155751755 +0000 UTC m=+2196.197799619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls") pod "maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" (UID: "883c3ad8-259f-436f-b48a-a825103617ad") : secret "maas-api-e2e-inf-b-d52a8f-serving-cert" not found
Jul 24 14:10:31.655880 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.655799    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.656476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.656419    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:31.666669 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:31.666636    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6v5\" (UniqueName: \"kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:32.160305 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.160263    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:32.162919 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.162886    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:32.304096 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.304055    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:10:32.307923 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.307900    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.310419 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.310396    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d52a8f\""
Jul 24 14:10:32.310772 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.310731    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d52a8f-dockercfg-mhkjz\""
Jul 24 14:10:32.320141 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.320112    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:10:32.345523 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.345480    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:32.404414 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.404380    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:10:32.409074 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.409049    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.431410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.431340    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:10:32.462757 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.462717    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.462954 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.462787    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q922\" (UniqueName: \"kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.517883 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.517851    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:10:32.518138 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:32.518113    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883c3ad8_259f_436f_b48a_a825103617ad.slice/crio-d560f3b3aa760bd7c43450d4fe2cceb2675497772d27a315abfef4a9d6ba7855 WatchSource:0}: Error finding container d560f3b3aa760bd7c43450d4fe2cceb2675497772d27a315abfef4a9d6ba7855: Status 404 returned error can't find the container with id d560f3b3aa760bd7c43450d4fe2cceb2675497772d27a315abfef4a9d6ba7855
Jul 24 14:10:32.563594 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.563556    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.563788 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.563648    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tm8\" (UniqueName: \"kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.563788 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.563681    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.563788 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.563707    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q922\" (UniqueName: \"kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.564360 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.564336    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.575070 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.575041    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q922\" (UniqueName: \"kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922\") pod \"payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.619831 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.619787    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:32.664759 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.664718    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.664929 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.664806    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tm8\" (UniqueName: \"kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.665521 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.665502    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.680017 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.679985    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tm8\" (UniqueName: \"kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8\") pod \"payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.723982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.723948    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:32.784545 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.784189    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:10:32.788098 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:32.788060    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e691be_52f9_455d_9952_454df0cfc89d.slice/crio-3f66367aaaf486d154f47bd605bd69497cb87f49ee4fd6af9d8a31c1af83432c WatchSource:0}: Error finding container 3f66367aaaf486d154f47bd605bd69497cb87f49ee4fd6af9d8a31c1af83432c: Status 404 returned error can't find the container with id 3f66367aaaf486d154f47bd605bd69497cb87f49ee4fd6af9d8a31c1af83432c
Jul 24 14:10:32.901569 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:32.901540    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:10:32.903376 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:10:32.903348    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a15f1e7_a462_4a33_8983_e60b548fef35.slice/crio-35ffe07c1bc629ec20ffafa0761c7bd4a0245567d9da5a3f37f433238c348e31 WatchSource:0}: Error finding container 35ffe07c1bc629ec20ffafa0761c7bd4a0245567d9da5a3f37f433238c348e31: Status 404 returned error can't find the container with id 35ffe07c1bc629ec20ffafa0761c7bd4a0245567d9da5a3f37f433238c348e31
Jul 24 14:10:33.342635 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.342597    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" event={"ID":"7a15f1e7-a462-4a33-8983-e60b548fef35","Type":"ContainerStarted","Data":"06d2d46fb5c56556cdaa8a643770d9a9844681dea7807de69b5458b14531bd62"}
Jul 24 14:10:33.343091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.342656    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" event={"ID":"7a15f1e7-a462-4a33-8983-e60b548fef35","Type":"ContainerStarted","Data":"35ffe07c1bc629ec20ffafa0761c7bd4a0245567d9da5a3f37f433238c348e31"}
Jul 24 14:10:33.343091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.342712    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:33.344329 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.344303    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" event={"ID":"74e691be-52f9-455d-9952-454df0cfc89d","Type":"ContainerStarted","Data":"c8539a72dd3538776b37e3d9852acb598bb05575af88a6ad69d9b5e2cbac7631"}
Jul 24 14:10:33.344456 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.344355    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:10:33.344456 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.344372    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" event={"ID":"74e691be-52f9-455d-9952-454df0cfc89d","Type":"ContainerStarted","Data":"3f66367aaaf486d154f47bd605bd69497cb87f49ee4fd6af9d8a31c1af83432c"}
Jul 24 14:10:33.345895 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.345870    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" event={"ID":"883c3ad8-259f-436f-b48a-a825103617ad","Type":"ContainerStarted","Data":"aa56cb3e8a3d54c58a5c73bc1452f4255274bd3b041a245433eca398cd05b715"}
Jul 24 14:10:33.345987 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.345917    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" event={"ID":"883c3ad8-259f-436f-b48a-a825103617ad","Type":"ContainerStarted","Data":"d560f3b3aa760bd7c43450d4fe2cceb2675497772d27a315abfef4a9d6ba7855"}
Jul 24 14:10:33.345987 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.345949    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:33.369640 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.369588    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" podStartSLOduration=1.369570727 podStartE2EDuration="1.369570727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:33.367166706 +0000 UTC m=+2197.409214590" watchObservedRunningTime="2026-07-24 14:10:33.369570727 +0000 UTC m=+2197.411618612"
Jul 24 14:10:33.396905 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:33.396838    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" podStartSLOduration=1.9091130010000001 podStartE2EDuration="2.396818201s" totalImagesPullingTime="487.7052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:33.394773837 +0000 UTC m=+2197.436821722" watchObservedRunningTime="2026-07-24 14:10:33.396818201 +0000 UTC m=+2197.438866092"
Jul 24 14:10:36.304832 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:36.304803    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:10:36.305297 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:36.304876    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:10:36.323754 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:36.323698    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" podStartSLOduration=4.323680891 podStartE2EDuration="4.323680891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:33.446243579 +0000 UTC m=+2197.488291464" watchObservedRunningTime="2026-07-24 14:10:36.323680891 +0000 UTC m=+2200.365728798"
Jul 24 14:10:39.356710 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:39.356682    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:10:44.353776 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:44.353730    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:10:44.354219 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:10:44.353952    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:12:02.212245 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.212211    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:12:02.212815 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.212415    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" podUID="fb09ed56-a69c-45bb-8977-d17dadc80570" containerName="maas-api" containerID="cri-o://5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f" gracePeriod=30
Jul 24 14:12:02.299892 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.299847    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:12:02.300102 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.300077    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" podUID="d61dddd9-cb45-4988-9621-3e8cc135579b" containerName="payload-processing" containerID="cri-o://2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5" gracePeriod=30
Jul 24 14:12:02.310202 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.310173    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:12:02.310394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.310374    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" podUID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" containerName="payload-pre-processing" containerID="cri-o://67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb" gracePeriod=30
Jul 24 14:12:02.572782 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.572748    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:12:02.624788 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.624762    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:12:02.628290 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.628266    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:12:02.696370 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696336    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpjwg\" (UniqueName: \"kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg\") pod \"fb09ed56-a69c-45bb-8977-d17dadc80570\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") "
Jul 24 14:12:02.696370 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696373    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c99ch\" (UniqueName: \"kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch\") pod \"d61dddd9-cb45-4988-9621-3e8cc135579b\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") "
Jul 24 14:12:02.696678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696408    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zj9p\" (UniqueName: \"kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p\") pod \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") "
Jul 24 14:12:02.696678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696495    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca\") pod \"fb09ed56-a69c-45bb-8977-d17dadc80570\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") "
Jul 24 14:12:02.696678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696577    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") pod \"fb09ed56-a69c-45bb-8977-d17dadc80570\" (UID: \"fb09ed56-a69c-45bb-8977-d17dadc80570\") "
Jul 24 14:12:02.696678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696599    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume\") pod \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\" (UID: \"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf\") "
Jul 24 14:12:02.696678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696620    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume\") pod \"d61dddd9-cb45-4988-9621-3e8cc135579b\" (UID: \"d61dddd9-cb45-4988-9621-3e8cc135579b\") "
Jul 24 14:12:02.696950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.696909    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb09ed56-a69c-45bb-8977-d17dadc80570" (UID: "fb09ed56-a69c-45bb-8977-d17dadc80570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:02.697036 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.697008    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" (UID: "5c7810a6-0f17-4fe8-b195-8c83b0dd87bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:02.697106 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.697032    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d61dddd9-cb45-4988-9621-3e8cc135579b" (UID: "d61dddd9-cb45-4988-9621-3e8cc135579b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:02.698888 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.698859    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb09ed56-a69c-45bb-8977-d17dadc80570" (UID: "fb09ed56-a69c-45bb-8977-d17dadc80570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:12:02.698888 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.698882    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch" (OuterVolumeSpecName: "kube-api-access-c99ch") pod "d61dddd9-cb45-4988-9621-3e8cc135579b" (UID: "d61dddd9-cb45-4988-9621-3e8cc135579b"). InnerVolumeSpecName "kube-api-access-c99ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:02.699052 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.698863    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p" (OuterVolumeSpecName: "kube-api-access-7zj9p") pod "5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" (UID: "5c7810a6-0f17-4fe8-b195-8c83b0dd87bf"). InnerVolumeSpecName "kube-api-access-7zj9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:02.699052 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.699044    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg" (OuterVolumeSpecName: "kube-api-access-rpjwg") pod "fb09ed56-a69c-45bb-8977-d17dadc80570" (UID: "fb09ed56-a69c-45bb-8977-d17dadc80570"). InnerVolumeSpecName "kube-api-access-rpjwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797780    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb09ed56-a69c-45bb-8977-d17dadc80570-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797813    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797827    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61dddd9-cb45-4988-9621-3e8cc135579b-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797839    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpjwg\" (UniqueName: \"kubernetes.io/projected/fb09ed56-a69c-45bb-8977-d17dadc80570-kube-api-access-rpjwg\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797852    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99ch\" (UniqueName: \"kubernetes.io/projected/d61dddd9-cb45-4988-9621-3e8cc135579b-kube-api-access-c99ch\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797863    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zj9p\" (UniqueName: \"kubernetes.io/projected/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf-kube-api-access-7zj9p\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.797882 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.797874    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb09ed56-a69c-45bb-8977-d17dadc80570-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:02.806958 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.806930    2527 generic.go:356] "Generic (PLEG): container finished" podID="fb09ed56-a69c-45bb-8977-d17dadc80570" containerID="5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f" exitCode=0
Jul 24 14:12:02.807152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.806973    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"
Jul 24 14:12:02.807152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.807012    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" event={"ID":"fb09ed56-a69c-45bb-8977-d17dadc80570","Type":"ContainerDied","Data":"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"}
Jul 24 14:12:02.807152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.807075    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn" event={"ID":"fb09ed56-a69c-45bb-8977-d17dadc80570","Type":"ContainerDied","Data":"8d9740b24002f5a9924835f886386e9f05019ae8dccb78c1e882e2102d9fe463"}
Jul 24 14:12:02.807152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.807105    2527 scope.go:117] "RemoveContainer" containerID="5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"
Jul 24 14:12:02.808341 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.808319    2527 generic.go:356] "Generic (PLEG): container finished" podID="d61dddd9-cb45-4988-9621-3e8cc135579b" containerID="2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5" exitCode=0
Jul 24 14:12:02.808535 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.808371    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"
Jul 24 14:12:02.808535 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.808398    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" event={"ID":"d61dddd9-cb45-4988-9621-3e8cc135579b","Type":"ContainerDied","Data":"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"}
Jul 24 14:12:02.808535 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.808479    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh" event={"ID":"d61dddd9-cb45-4988-9621-3e8cc135579b","Type":"ContainerDied","Data":"5a6be107b08072e992415955841ed90d277fb2b6345f833d75960b2a85eae4fa"}
Jul 24 14:12:02.810412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.809843    2527 generic.go:356] "Generic (PLEG): container finished" podID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" containerID="67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb" exitCode=0
Jul 24 14:12:02.810412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.809884    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" event={"ID":"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf","Type":"ContainerDied","Data":"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"}
Jul 24 14:12:02.810412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.809933    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf" event={"ID":"5c7810a6-0f17-4fe8-b195-8c83b0dd87bf","Type":"ContainerDied","Data":"8aab6ba67308c3df2676deea0b0193d930dec761c022b320d1634994b97cb9c8"}
Jul 24 14:12:02.810412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.809892    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"
Jul 24 14:12:02.816961 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.816931    2527 scope.go:117] "RemoveContainer" containerID="5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"
Jul 24 14:12:02.817269 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:12:02.817246    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f\": container with ID starting with 5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f not found: ID does not exist" containerID="5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"
Jul 24 14:12:02.817349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.817284    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f"} err="failed to get container status \"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f\": rpc error: code = NotFound desc = could not find container \"5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f\": container with ID starting with 5ff208a4ea7153ffca8b7cf95a8407604f0476d526a07fdbd792a210f12e922f not found: ID does not exist"
Jul 24 14:12:02.817349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.817307    2527 scope.go:117] "RemoveContainer" containerID="2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"
Jul 24 14:12:02.826583 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.826565    2527 scope.go:117] "RemoveContainer" containerID="2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"
Jul 24 14:12:02.826867 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:12:02.826849    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5\": container with ID starting with 2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5 not found: ID does not exist" containerID="2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"
Jul 24 14:12:02.826930 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.826872    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5"} err="failed to get container status \"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5\": rpc error: code = NotFound desc = could not find container \"2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5\": container with ID starting with 2a07dc2967c8c08e3c65e66ddf78ff10ec5b2f1be684046e4880936ae6d450a5 not found: ID does not exist"
Jul 24 14:12:02.826930 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.826891    2527 scope.go:117] "RemoveContainer" containerID="67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"
Jul 24 14:12:02.836021 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.835992    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:12:02.836401 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.836389    2527 scope.go:117] "RemoveContainer" containerID="67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"
Jul 24 14:12:02.836706 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:12:02.836681    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb\": container with ID starting with 67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb not found: ID does not exist" containerID="67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"
Jul 24 14:12:02.836761 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.836719    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb"} err="failed to get container status \"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb\": rpc error: code = NotFound desc = could not find container \"67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb\": container with ID starting with 67da7c3b4605e7f249c03f8c4202d38964706514693dac749e297c4997d853fb not found: ID does not exist"
Jul 24 14:12:02.839690 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.839642    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c21e7d-5cb55ddf76-njjlf"]
Jul 24 14:12:02.851993 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.851959    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:12:02.858820 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.858794    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c21e7d-dcc6c6b-5twwn"]
Jul 24 14:12:02.875671 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.875555    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:12:02.878393 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:02.878364    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c21e7d-7754cdf998-7m9jh"]
Jul 24 14:12:04.514654 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:04.514621    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" path="/var/lib/kubelet/pods/5c7810a6-0f17-4fe8-b195-8c83b0dd87bf/volumes"
Jul 24 14:12:04.515034 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:04.514988    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d61dddd9-cb45-4988-9621-3e8cc135579b" path="/var/lib/kubelet/pods/d61dddd9-cb45-4988-9621-3e8cc135579b/volumes"
Jul 24 14:12:04.515319 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:04.515306    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb09ed56-a69c-45bb-8977-d17dadc80570" path="/var/lib/kubelet/pods/fb09ed56-a69c-45bb-8977-d17dadc80570/volumes"
Jul 24 14:12:24.703378 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.703339    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:12:24.703900 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.703623    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" podUID="883c3ad8-259f-436f-b48a-a825103617ad" containerName="maas-api" containerID="cri-o://aa56cb3e8a3d54c58a5c73bc1452f4255274bd3b041a245433eca398cd05b715" gracePeriod=30
Jul 24 14:12:24.775915 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.775881    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:12:24.776162 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.776131    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" podUID="7a15f1e7-a462-4a33-8983-e60b548fef35" containerName="payload-processing" containerID="cri-o://06d2d46fb5c56556cdaa8a643770d9a9844681dea7807de69b5458b14531bd62" gracePeriod=30
Jul 24 14:12:24.789023 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.788989    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:12:24.789239 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.789218    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" podUID="74e691be-52f9-455d-9952-454df0cfc89d" containerName="payload-pre-processing" containerID="cri-o://c8539a72dd3538776b37e3d9852acb598bb05575af88a6ad69d9b5e2cbac7631" gracePeriod=30
Jul 24 14:12:24.942333 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.942291    2527 generic.go:356] "Generic (PLEG): container finished" podID="7a15f1e7-a462-4a33-8983-e60b548fef35" containerID="06d2d46fb5c56556cdaa8a643770d9a9844681dea7807de69b5458b14531bd62" exitCode=0
Jul 24 14:12:24.942333 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.942310    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" event={"ID":"7a15f1e7-a462-4a33-8983-e60b548fef35","Type":"ContainerDied","Data":"06d2d46fb5c56556cdaa8a643770d9a9844681dea7807de69b5458b14531bd62"}
Jul 24 14:12:24.944410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.944378    2527 generic.go:356] "Generic (PLEG): container finished" podID="74e691be-52f9-455d-9952-454df0cfc89d" containerID="c8539a72dd3538776b37e3d9852acb598bb05575af88a6ad69d9b5e2cbac7631" exitCode=0
Jul 24 14:12:24.944559 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.944538    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" event={"ID":"74e691be-52f9-455d-9952-454df0cfc89d","Type":"ContainerDied","Data":"c8539a72dd3538776b37e3d9852acb598bb05575af88a6ad69d9b5e2cbac7631"}
Jul 24 14:12:24.946472 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.946420    2527 generic.go:356] "Generic (PLEG): container finished" podID="883c3ad8-259f-436f-b48a-a825103617ad" containerID="aa56cb3e8a3d54c58a5c73bc1452f4255274bd3b041a245433eca398cd05b715" exitCode=0
Jul 24 14:12:24.946599 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:24.946484    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" event={"ID":"883c3ad8-259f-436f-b48a-a825103617ad","Type":"ContainerDied","Data":"aa56cb3e8a3d54c58a5c73bc1452f4255274bd3b041a245433eca398cd05b715"}
Jul 24 14:12:25.083989 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.083962    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:12:25.091765 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.091736    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:12:25.099937 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.099913    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:12:25.203288 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203254    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume\") pod \"74e691be-52f9-455d-9952-454df0cfc89d\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") "
Jul 24 14:12:25.203288 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203293    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt6v5\" (UniqueName: \"kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5\") pod \"883c3ad8-259f-436f-b48a-a825103617ad\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") "
Jul 24 14:12:25.203602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203311    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume\") pod \"7a15f1e7-a462-4a33-8983-e60b548fef35\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") "
Jul 24 14:12:25.203602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203362    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca\") pod \"883c3ad8-259f-436f-b48a-a825103617ad\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") "
Jul 24 14:12:25.203602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203402    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q922\" (UniqueName: \"kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922\") pod \"74e691be-52f9-455d-9952-454df0cfc89d\" (UID: \"74e691be-52f9-455d-9952-454df0cfc89d\") "
Jul 24 14:12:25.203602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203475    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42tm8\" (UniqueName: \"kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8\") pod \"7a15f1e7-a462-4a33-8983-e60b548fef35\" (UID: \"7a15f1e7-a462-4a33-8983-e60b548fef35\") "
Jul 24 14:12:25.203602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203532    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") pod \"883c3ad8-259f-436f-b48a-a825103617ad\" (UID: \"883c3ad8-259f-436f-b48a-a825103617ad\") "
Jul 24 14:12:25.203873 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203778    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74e691be-52f9-455d-9952-454df0cfc89d" (UID: "74e691be-52f9-455d-9952-454df0cfc89d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:25.203873 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203804    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "883c3ad8-259f-436f-b48a-a825103617ad" (UID: "883c3ad8-259f-436f-b48a-a825103617ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:25.203873 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203821    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a15f1e7-a462-4a33-8983-e60b548fef35" (UID: "7a15f1e7-a462-4a33-8983-e60b548fef35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:25.204044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203930    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e691be-52f9-455d-9952-454df0cfc89d-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.204044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203949    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a15f1e7-a462-4a33-8983-e60b548fef35-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.204044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.203963    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883c3ad8-259f-436f-b48a-a825103617ad-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.205774 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.205747    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922" (OuterVolumeSpecName: "kube-api-access-9q922") pod "74e691be-52f9-455d-9952-454df0cfc89d" (UID: "74e691be-52f9-455d-9952-454df0cfc89d"). InnerVolumeSpecName "kube-api-access-9q922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:25.205774 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.205767    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5" (OuterVolumeSpecName: "kube-api-access-dt6v5") pod "883c3ad8-259f-436f-b48a-a825103617ad" (UID: "883c3ad8-259f-436f-b48a-a825103617ad"). InnerVolumeSpecName "kube-api-access-dt6v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:25.205925 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.205764    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8" (OuterVolumeSpecName: "kube-api-access-42tm8") pod "7a15f1e7-a462-4a33-8983-e60b548fef35" (UID: "7a15f1e7-a462-4a33-8983-e60b548fef35"). InnerVolumeSpecName "kube-api-access-42tm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:25.205987 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.205955    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "883c3ad8-259f-436f-b48a-a825103617ad" (UID: "883c3ad8-259f-436f-b48a-a825103617ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:12:25.304982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.304896    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q922\" (UniqueName: \"kubernetes.io/projected/74e691be-52f9-455d-9952-454df0cfc89d-kube-api-access-9q922\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.304982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.304929    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tm8\" (UniqueName: \"kubernetes.io/projected/7a15f1e7-a462-4a33-8983-e60b548fef35-kube-api-access-42tm8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.304982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.304939    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883c3ad8-259f-436f-b48a-a825103617ad-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.304982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.304948    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6v5\" (UniqueName: \"kubernetes.io/projected/883c3ad8-259f-436f-b48a-a825103617ad-kube-api-access-dt6v5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:12:25.953250 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.953216    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"
Jul 24 14:12:25.953763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.953248    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn" event={"ID":"883c3ad8-259f-436f-b48a-a825103617ad","Type":"ContainerDied","Data":"d560f3b3aa760bd7c43450d4fe2cceb2675497772d27a315abfef4a9d6ba7855"}
Jul 24 14:12:25.953763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.953306    2527 scope.go:117] "RemoveContainer" containerID="aa56cb3e8a3d54c58a5c73bc1452f4255274bd3b041a245433eca398cd05b715"
Jul 24 14:12:25.954856 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.954822    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq" event={"ID":"7a15f1e7-a462-4a33-8983-e60b548fef35","Type":"ContainerDied","Data":"35ffe07c1bc629ec20ffafa0761c7bd4a0245567d9da5a3f37f433238c348e31"}
Jul 24 14:12:25.954993 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.954828    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"
Jul 24 14:12:25.956170 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.956152    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"
Jul 24 14:12:25.956309 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.956148    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs" event={"ID":"74e691be-52f9-455d-9952-454df0cfc89d","Type":"ContainerDied","Data":"3f66367aaaf486d154f47bd605bd69497cb87f49ee4fd6af9d8a31c1af83432c"}
Jul 24 14:12:25.968716 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.968634    2527 scope.go:117] "RemoveContainer" containerID="06d2d46fb5c56556cdaa8a643770d9a9844681dea7807de69b5458b14531bd62"
Jul 24 14:12:25.978960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.978939    2527 scope.go:117] "RemoveContainer" containerID="c8539a72dd3538776b37e3d9852acb598bb05575af88a6ad69d9b5e2cbac7631"
Jul 24 14:12:25.991422 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.991392    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:12:25.996338 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:25.996312    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d52a8f-575cf7588f-hx4jn"]
Jul 24 14:12:26.007988 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.007959    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:12:26.010329 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.010300    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d52a8f-5d695c968d-fcrgs"]
Jul 24 14:12:26.022965 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.022936    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:12:26.028859 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.028830    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d52a8f-5fd55cfbb4-pz2zq"]
Jul 24 14:12:26.513776 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.513742    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e691be-52f9-455d-9952-454df0cfc89d" path="/var/lib/kubelet/pods/74e691be-52f9-455d-9952-454df0cfc89d/volumes"
Jul 24 14:12:26.514155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.514141    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a15f1e7-a462-4a33-8983-e60b548fef35" path="/var/lib/kubelet/pods/7a15f1e7-a462-4a33-8983-e60b548fef35/volumes"
Jul 24 14:12:26.514565 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:26.514552    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883c3ad8-259f-436f-b48a-a825103617ad" path="/var/lib/kubelet/pods/883c3ad8-259f-436f-b48a-a825103617ad/volumes"
Jul 24 14:12:40.017978 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.017940    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018340    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a15f1e7-a462-4a33-8983-e60b548fef35" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018355    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a15f1e7-a462-4a33-8983-e60b548fef35" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018373    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e691be-52f9-455d-9952-454df0cfc89d" containerName="payload-pre-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018379    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e691be-52f9-455d-9952-454df0cfc89d" containerName="payload-pre-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018392    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" containerName="payload-pre-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018397    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" containerName="payload-pre-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018403    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb09ed56-a69c-45bb-8977-d17dadc80570" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018408    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb09ed56-a69c-45bb-8977-d17dadc80570" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018414    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61dddd9-cb45-4988-9621-3e8cc135579b" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018419    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61dddd9-cb45-4988-9621-3e8cc135579b" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018444    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883c3ad8-259f-436f-b48a-a825103617ad" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018448    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="883c3ad8-259f-436f-b48a-a825103617ad" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018509    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61dddd9-cb45-4988-9621-3e8cc135579b" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018529    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="883c3ad8-259f-436f-b48a-a825103617ad" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018537    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb09ed56-a69c-45bb-8977-d17dadc80570" containerName="maas-api"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018543    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e691be-52f9-455d-9952-454df0cfc89d" containerName="payload-pre-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018548    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a15f1e7-a462-4a33-8983-e60b548fef35" containerName="payload-processing"
Jul 24 14:12:40.018564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.018554    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7810a6-0f17-4fe8-b195-8c83b0dd87bf" containerName="payload-pre-processing"
Jul 24 14:12:40.021853 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.021833    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.024180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.024149    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-41b861-serving-cert\""
Jul 24 14:12:40.036786 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.036757    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:12:40.124950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.124911    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.124950 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.124955    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkxb\" (UniqueName: \"kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.125223 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.125073    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.226132 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.226087    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.226337 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.226150    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkxb\" (UniqueName: \"kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.226337 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:12:40.226208    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-serving-cert: secret "maas-api-e2e-api-a-41b861-serving-cert" not found
Jul 24 14:12:40.226337 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.226271    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.226337 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:12:40.226296    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls podName:cbdc4415-e609-4a2c-840f-5ed87a0663a7 nodeName:}" failed. No retries permitted until 2026-07-24 14:12:40.726274562 +0000 UTC m=+2324.768322432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls") pod "maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" (UID: "cbdc4415-e609-4a2c-840f-5ed87a0663a7") : secret "maas-api-e2e-api-a-41b861-serving-cert" not found
Jul 24 14:12:40.226988 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.226969    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.245038 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.245008    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkxb\" (UniqueName: \"kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.731220 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.731180    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.733961 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.733932    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") pod \"maas-api-e2e-api-a-41b861-7b8b896995-gr6cx\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:40.933146 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:40.933098    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:41.053625 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.053577    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:12:41.058144 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.058112    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.060897 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.060873    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-41b861\""
Jul 24 14:12:41.061070 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.060903    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-41b861-dockercfg-qvknt\""
Jul 24 14:12:41.074582 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.074553    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:12:41.103637 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.103607    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:12:41.106204 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:12:41.106177    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdc4415_e609_4a2c_840f_5ed87a0663a7.slice/crio-735e10d885ec382ae7a4817f67855e63a0b9a09fbbeea3fce3aa503a8a5187ae WatchSource:0}: Error finding container 735e10d885ec382ae7a4817f67855e63a0b9a09fbbeea3fce3aa503a8a5187ae: Status 404 returned error can't find the container with id 735e10d885ec382ae7a4817f67855e63a0b9a09fbbeea3fce3aa503a8a5187ae
Jul 24 14:12:41.106788 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.106766    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:12:41.107483 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.107462    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:12:41.110573 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.110551    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.125851 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.125823    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:12:41.135909 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.135877    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzkp\" (UniqueName: \"kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.136075 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.135917    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dmp\" (UniqueName: \"kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.136075 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.135950    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.136075 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.136053    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.237008 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.236964    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzkp\" (UniqueName: \"kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.237008 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.237010    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dmp\" (UniqueName: \"kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.237244 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.237039    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.237244 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.237086    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.238035 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.238010    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.238155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.238048    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.246311 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.246234    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzkp\" (UniqueName: \"kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp\") pod \"payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") " pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.246311 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.246298    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dmp\" (UniqueName: \"kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp\") pod \"payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.372095 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.372055    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:41.424619 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.424517    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:41.537349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.537312    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:12:41.538368 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:12:41.538337    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f5e97b_1546_4063_856a_7d3b611e3eb6.slice/crio-fec47ff289fffd29ca020c1c63a789bd379942b4b5a84b7bbdf902189d0f4d5e WatchSource:0}: Error finding container fec47ff289fffd29ca020c1c63a789bd379942b4b5a84b7bbdf902189d0f4d5e: Status 404 returned error can't find the container with id fec47ff289fffd29ca020c1c63a789bd379942b4b5a84b7bbdf902189d0f4d5e
Jul 24 14:12:41.595530 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:41.595507    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:12:41.597709 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:12:41.597684    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beeb7c6_619e_4cd9_86f9_9b2ffc868ee2.slice/crio-453e669223fdfe848df274ecd6298ef411cede326d85aafac9c07db7cf75bf64 WatchSource:0}: Error finding container 453e669223fdfe848df274ecd6298ef411cede326d85aafac9c07db7cf75bf64: Status 404 returned error can't find the container with id 453e669223fdfe848df274ecd6298ef411cede326d85aafac9c07db7cf75bf64
Jul 24 14:12:42.045206 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.045163    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" event={"ID":"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2","Type":"ContainerStarted","Data":"74b7ba7ecf4d2487539cbae6f263367398f014203267253d734235948088517d"}
Jul 24 14:12:42.045413 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.045230    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" event={"ID":"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2","Type":"ContainerStarted","Data":"453e669223fdfe848df274ecd6298ef411cede326d85aafac9c07db7cf75bf64"}
Jul 24 14:12:42.045413 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.045344    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:12:42.046840 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.046815    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" event={"ID":"cbdc4415-e609-4a2c-840f-5ed87a0663a7","Type":"ContainerStarted","Data":"05c7ceebb1cb96c75f46bfaf6705cc9b23c0346c6cd18ab6a3918e55172e0fce"}
Jul 24 14:12:42.046959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.046862    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" event={"ID":"cbdc4415-e609-4a2c-840f-5ed87a0663a7","Type":"ContainerStarted","Data":"735e10d885ec382ae7a4817f67855e63a0b9a09fbbeea3fce3aa503a8a5187ae"}
Jul 24 14:12:42.046959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.046888    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:42.048338 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.048313    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" event={"ID":"d4f5e97b-1546-4063-856a-7d3b611e3eb6","Type":"ContainerStarted","Data":"eaaadd5534d97261ee69251d1f99aa788a9e9f4d635a5d7dffd120cdac311355"}
Jul 24 14:12:42.048455 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.048357    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" event={"ID":"d4f5e97b-1546-4063-856a-7d3b611e3eb6","Type":"ContainerStarted","Data":"fec47ff289fffd29ca020c1c63a789bd379942b4b5a84b7bbdf902189d0f4d5e"}
Jul 24 14:12:42.048509 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.048457    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:42.065297 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.065244    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" podStartSLOduration=1.065228663 podStartE2EDuration="1.065228663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:42.063296906 +0000 UTC m=+2326.105344791" watchObservedRunningTime="2026-07-24 14:12:42.065228663 +0000 UTC m=+2326.107276549"
Jul 24 14:12:42.082866 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.082808    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" podStartSLOduration=1.082791943 podStartE2EDuration="1.082791943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:42.079974678 +0000 UTC m=+2326.122022563" watchObservedRunningTime="2026-07-24 14:12:42.082791943 +0000 UTC m=+2326.124839828"
Jul 24 14:12:42.106535 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:42.106467    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" podStartSLOduration=2.64323197 podStartE2EDuration="3.106427367s" totalImagesPullingTime="463.195397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:42.104824533 +0000 UTC m=+2326.146872418" watchObservedRunningTime="2026-07-24 14:12:42.106427367 +0000 UTC m=+2326.148475252"
Jul 24 14:12:48.058490 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.058392    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:12:48.093757 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.093718    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:12:48.099021 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.098993    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.101524 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.101405    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-17b50a-serving-cert\""
Jul 24 14:12:48.115094 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.115060    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:12:48.202204 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.202170    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.202410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.202217    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkv49\" (UniqueName: \"kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.202410 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.202305    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.303356 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.303315    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.303608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.303370    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkv49\" (UniqueName: \"kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.303608 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.303483    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.304247 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.304173    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.306219 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.306192    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.311845 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.311779    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkv49\" (UniqueName: \"kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49\") pod \"maas-api-e2e-api-b-17b50a-678867df78-zn95m\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.413652 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.413614    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:48.571105 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:48.570997    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:12:48.574121 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:12:48.574092    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08508308_cbb1_4c5f_bac4_23e7d6187f02.slice/crio-4ad61831162f2708b77a6c83605a6a42d1798ce10ae46cfbbdc9cdbe9454ee02 WatchSource:0}: Error finding container 4ad61831162f2708b77a6c83605a6a42d1798ce10ae46cfbbdc9cdbe9454ee02: Status 404 returned error can't find the container with id 4ad61831162f2708b77a6c83605a6a42d1798ce10ae46cfbbdc9cdbe9454ee02
Jul 24 14:12:49.088512 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:49.088472    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" event={"ID":"08508308-cbb1-4c5f-bac4-23e7d6187f02","Type":"ContainerStarted","Data":"8ce6df67af0004e66712728ba8fbf182a441588cda8e8822c429067ed60a0064"}
Jul 24 14:12:49.088935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:49.088537    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" event={"ID":"08508308-cbb1-4c5f-bac4-23e7d6187f02","Type":"ContainerStarted","Data":"4ad61831162f2708b77a6c83605a6a42d1798ce10ae46cfbbdc9cdbe9454ee02"}
Jul 24 14:12:49.088935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:49.088579    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:12:49.109307 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:49.109247    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" podStartSLOduration=0.802470731 podStartE2EDuration="1.109224194s" totalImagesPullingTime="306.753463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:49.108985459 +0000 UTC m=+2333.151033346" watchObservedRunningTime="2026-07-24 14:12:49.109224194 +0000 UTC m=+2333.151272081"
Jul 24 14:12:53.055660 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:53.055609    2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.72:9004: connect: connection refused"
Jul 24 14:12:53.056128 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:53.056036    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:12:55.100951 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:12:55.100921    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:13:03.054602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:03.054572    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:13:16.016168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.016131    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:13:16.016737 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.016393    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" podUID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" containerName="maas-api" containerID="cri-o://05c7ceebb1cb96c75f46bfaf6705cc9b23c0346c6cd18ab6a3918e55172e0fce" gracePeriod=30
Jul 24 14:13:16.095394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.095354    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:13:16.095619 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.095589    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" podUID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" containerName="payload-processing" containerID="cri-o://eaaadd5534d97261ee69251d1f99aa788a9e9f4d635a5d7dffd120cdac311355" gracePeriod=30
Jul 24 14:13:16.106206 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.106172    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:13:16.106408 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.106381    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerName="payload-pre-processing" containerID="cri-o://74b7ba7ecf4d2487539cbae6f263367398f014203267253d734235948088517d" gracePeriod=30
Jul 24 14:13:16.234175 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.234133    2527 generic.go:356] "Generic (PLEG): container finished" podID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerID="74b7ba7ecf4d2487539cbae6f263367398f014203267253d734235948088517d" exitCode=0
Jul 24 14:13:16.234368 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.234200    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" event={"ID":"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2","Type":"ContainerDied","Data":"74b7ba7ecf4d2487539cbae6f263367398f014203267253d734235948088517d"}
Jul 24 14:13:16.236245 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.236216    2527 generic.go:356] "Generic (PLEG): container finished" podID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" containerID="05c7ceebb1cb96c75f46bfaf6705cc9b23c0346c6cd18ab6a3918e55172e0fce" exitCode=0
Jul 24 14:13:16.236404 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.236331    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" event={"ID":"cbdc4415-e609-4a2c-840f-5ed87a0663a7","Type":"ContainerDied","Data":"05c7ceebb1cb96c75f46bfaf6705cc9b23c0346c6cd18ab6a3918e55172e0fce"}
Jul 24 14:13:16.238281 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.238211    2527 generic.go:356] "Generic (PLEG): container finished" podID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" containerID="eaaadd5534d97261ee69251d1f99aa788a9e9f4d635a5d7dffd120cdac311355" exitCode=0
Jul 24 14:13:16.238461 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.238307    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" event={"ID":"d4f5e97b-1546-4063-856a-7d3b611e3eb6","Type":"ContainerDied","Data":"eaaadd5534d97261ee69251d1f99aa788a9e9f4d635a5d7dffd120cdac311355"}
Jul 24 14:13:16.378379 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.378341    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:13:16.403141 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.403110    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:13:16.413515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.413396    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:13:16.463848 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.463813    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") pod \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") "
Jul 24 14:13:16.464041 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.463868    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca\") pod \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") "
Jul 24 14:13:16.464041 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.463937    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zkxb\" (UniqueName: \"kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb\") pod \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\" (UID: \"cbdc4415-e609-4a2c-840f-5ed87a0663a7\") "
Jul 24 14:13:16.464263 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.464235    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbdc4415-e609-4a2c-840f-5ed87a0663a7" (UID: "cbdc4415-e609-4a2c-840f-5ed87a0663a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:16.466189 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.466161    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb" (OuterVolumeSpecName: "kube-api-access-9zkxb") pod "cbdc4415-e609-4a2c-840f-5ed87a0663a7" (UID: "cbdc4415-e609-4a2c-840f-5ed87a0663a7"). InnerVolumeSpecName "kube-api-access-9zkxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:16.466329 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.466192    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbdc4415-e609-4a2c-840f-5ed87a0663a7" (UID: "cbdc4415-e609-4a2c-840f-5ed87a0663a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:13:16.565377 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565278    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2dmp\" (UniqueName: \"kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp\") pod \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") "
Jul 24 14:13:16.565377 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565336    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtzkp\" (UniqueName: \"kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp\") pod \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") "
Jul 24 14:13:16.565377 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565357    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume\") pod \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\" (UID: \"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2\") "
Jul 24 14:13:16.565729 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565394    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume\") pod \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\" (UID: \"d4f5e97b-1546-4063-856a-7d3b611e3eb6\") "
Jul 24 14:13:16.565791 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565745    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zkxb\" (UniqueName: \"kubernetes.io/projected/cbdc4415-e609-4a2c-840f-5ed87a0663a7-kube-api-access-9zkxb\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.565791 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565768    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbdc4415-e609-4a2c-840f-5ed87a0663a7-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.565791 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565782    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc4415-e609-4a2c-840f-5ed87a0663a7-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.565912 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565802    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" (UID: "3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:16.565912 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.565823    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4f5e97b-1546-4063-856a-7d3b611e3eb6" (UID: "d4f5e97b-1546-4063-856a-7d3b611e3eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:16.567747 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.567721    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp" (OuterVolumeSpecName: "kube-api-access-rtzkp") pod "d4f5e97b-1546-4063-856a-7d3b611e3eb6" (UID: "d4f5e97b-1546-4063-856a-7d3b611e3eb6"). InnerVolumeSpecName "kube-api-access-rtzkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:16.567822 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.567799    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp" (OuterVolumeSpecName: "kube-api-access-m2dmp") pod "3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" (UID: "3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2"). InnerVolumeSpecName "kube-api-access-m2dmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:16.667091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.667046    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2dmp\" (UniqueName: \"kubernetes.io/projected/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-kube-api-access-m2dmp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.667091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.667080    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzkp\" (UniqueName: \"kubernetes.io/projected/d4f5e97b-1546-4063-856a-7d3b611e3eb6-kube-api-access-rtzkp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.667091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.667090    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:16.667091 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:16.667098    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f5e97b-1546-4063-856a-7d3b611e3eb6-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:17.244934 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.244904    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"
Jul 24 14:13:17.245417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.244927    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh" event={"ID":"d4f5e97b-1546-4063-856a-7d3b611e3eb6","Type":"ContainerDied","Data":"fec47ff289fffd29ca020c1c63a789bd379942b4b5a84b7bbdf902189d0f4d5e"}
Jul 24 14:13:17.245417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.245026    2527 scope.go:117] "RemoveContainer" containerID="eaaadd5534d97261ee69251d1f99aa788a9e9f4d635a5d7dffd120cdac311355"
Jul 24 14:13:17.246579 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.246550    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt" event={"ID":"3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2","Type":"ContainerDied","Data":"453e669223fdfe848df274ecd6298ef411cede326d85aafac9c07db7cf75bf64"}
Jul 24 14:13:17.246692 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.246559    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"
Jul 24 14:13:17.248346 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.248308    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx" event={"ID":"cbdc4415-e609-4a2c-840f-5ed87a0663a7","Type":"ContainerDied","Data":"735e10d885ec382ae7a4817f67855e63a0b9a09fbbeea3fce3aa503a8a5187ae"}
Jul 24 14:13:17.248492 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.248355    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"
Jul 24 14:13:17.255562 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.255544    2527 scope.go:117] "RemoveContainer" containerID="74b7ba7ecf4d2487539cbae6f263367398f014203267253d734235948088517d"
Jul 24 14:13:17.264375 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.264354    2527 scope.go:117] "RemoveContainer" containerID="05c7ceebb1cb96c75f46bfaf6705cc9b23c0346c6cd18ab6a3918e55172e0fce"
Jul 24 14:13:17.283166 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.283081    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:13:17.287366 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.287336    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-41b861-84cb4964dd-4ksjh"]
Jul 24 14:13:17.308629 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.308595    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:13:17.314765 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.314734    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-41b861-7b8b896995-gr6cx"]
Jul 24 14:13:17.334427 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.334360    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:13:17.337656 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:17.337626    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-41b861-6cbc69d864-jfppt"]
Jul 24 14:13:18.515660 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:18.515628    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" path="/var/lib/kubelet/pods/3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2/volumes"
Jul 24 14:13:18.516054 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:18.516022    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" path="/var/lib/kubelet/pods/cbdc4415-e609-4a2c-840f-5ed87a0663a7/volumes"
Jul 24 14:13:18.516330 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:18.516316    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" path="/var/lib/kubelet/pods/d4f5e97b-1546-4063-856a-7d3b611e3eb6/volumes"
Jul 24 14:13:44.185110 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.185068    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:13:44.186236 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.186134    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" podUID="08508308-cbb1-4c5f-bac4-23e7d6187f02" containerName="maas-api" containerID="cri-o://8ce6df67af0004e66712728ba8fbf182a441588cda8e8822c429067ed60a0064" gracePeriod=30
Jul 24 14:13:44.394695 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.390247    2527 generic.go:356] "Generic (PLEG): container finished" podID="08508308-cbb1-4c5f-bac4-23e7d6187f02" containerID="8ce6df67af0004e66712728ba8fbf182a441588cda8e8822c429067ed60a0064" exitCode=0
Jul 24 14:13:44.394695 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.390335    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" event={"ID":"08508308-cbb1-4c5f-bac4-23e7d6187f02","Type":"ContainerDied","Data":"8ce6df67af0004e66712728ba8fbf182a441588cda8e8822c429067ed60a0064"}
Jul 24 14:13:44.471711 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.471678    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:13:44.521480 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.521447    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkv49\" (UniqueName: \"kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49\") pod \"08508308-cbb1-4c5f-bac4-23e7d6187f02\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") "
Jul 24 14:13:44.521667 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.521493    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls\") pod \"08508308-cbb1-4c5f-bac4-23e7d6187f02\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") "
Jul 24 14:13:44.521667 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.521561    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca\") pod \"08508308-cbb1-4c5f-bac4-23e7d6187f02\" (UID: \"08508308-cbb1-4c5f-bac4-23e7d6187f02\") "
Jul 24 14:13:44.522103 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.522072    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08508308-cbb1-4c5f-bac4-23e7d6187f02" (UID: "08508308-cbb1-4c5f-bac4-23e7d6187f02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:44.523897 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.523866    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08508308-cbb1-4c5f-bac4-23e7d6187f02" (UID: "08508308-cbb1-4c5f-bac4-23e7d6187f02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:13:44.523998 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.523915    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49" (OuterVolumeSpecName: "kube-api-access-hkv49") pod "08508308-cbb1-4c5f-bac4-23e7d6187f02" (UID: "08508308-cbb1-4c5f-bac4-23e7d6187f02"). InnerVolumeSpecName "kube-api-access-hkv49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:44.622231 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.622194    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkv49\" (UniqueName: \"kubernetes.io/projected/08508308-cbb1-4c5f-bac4-23e7d6187f02-kube-api-access-hkv49\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.622231 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.622225    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08508308-cbb1-4c5f-bac4-23e7d6187f02-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.622231 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:44.622234    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08508308-cbb1-4c5f-bac4-23e7d6187f02-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:13:45.396518 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:45.396480    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m" event={"ID":"08508308-cbb1-4c5f-bac4-23e7d6187f02","Type":"ContainerDied","Data":"4ad61831162f2708b77a6c83605a6a42d1798ce10ae46cfbbdc9cdbe9454ee02"}
Jul 24 14:13:45.396935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:45.396541    2527 scope.go:117] "RemoveContainer" containerID="8ce6df67af0004e66712728ba8fbf182a441588cda8e8822c429067ed60a0064"
Jul 24 14:13:45.396935 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:45.396497    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"
Jul 24 14:13:45.421852 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:45.421820    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:13:45.427125 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:45.427093    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-17b50a-678867df78-zn95m"]
Jul 24 14:13:46.514383 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:46.514342    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08508308-cbb1-4c5f-bac4-23e7d6187f02" path="/var/lib/kubelet/pods/08508308-cbb1-4c5f-bac4-23e7d6187f02/volumes"
Jul 24 14:13:56.562400 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:56.562368    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:13:56.564398 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:56.564376    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:13:59.563643 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.563605    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:13:59.564175 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564159    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerName="payload-pre-processing"
Jul 24 14:13:59.564225 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564179    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerName="payload-pre-processing"
Jul 24 14:13:59.564225 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564215    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" containerName="payload-processing"
Jul 24 14:13:59.564291 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564224    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" containerName="payload-processing"
Jul 24 14:13:59.564291 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564237    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" containerName="maas-api"
Jul 24 14:13:59.564291 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564248    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" containerName="maas-api"
Jul 24 14:13:59.564291 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564259    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08508308-cbb1-4c5f-bac4-23e7d6187f02" containerName="maas-api"
Jul 24 14:13:59.564291 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564267    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="08508308-cbb1-4c5f-bac4-23e7d6187f02" containerName="maas-api"
Jul 24 14:13:59.564457 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564353    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beeb7c6-619e-4cd9-86f9-9b2ffc868ee2" containerName="payload-pre-processing"
Jul 24 14:13:59.564457 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564368    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="08508308-cbb1-4c5f-bac4-23e7d6187f02" containerName="maas-api"
Jul 24 14:13:59.564457 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564379    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbdc4415-e609-4a2c-840f-5ed87a0663a7" containerName="maas-api"
Jul 24 14:13:59.564457 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.564392    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f5e97b-1546-4063-856a-7d3b611e3eb6" containerName="payload-processing"
Jul 24 14:13:59.569394 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.569370    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.571476 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.571449    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4cd364-serving-cert\""
Jul 24 14:13:59.578988 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.578964    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:13:59.660539 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.660496    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.660539 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.660533    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2s7n\" (UniqueName: \"kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.660830 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.660605    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.761556 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.761514    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.761556 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.761553    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2s7n\" (UniqueName: \"kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.761825 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.761578    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.761825 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:13:59.761682    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-serving-cert: secret "maas-api-e2e-shared-a-4cd364-serving-cert" not found
Jul 24 14:13:59.761825 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:13:59.761795    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls podName:84989689-958d-4f13-8bb6-eff963fc2b72 nodeName:}" failed. No retries permitted until 2026-07-24 14:14:00.261771378 +0000 UTC m=+2404.303819242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls") pod "maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" (UID: "84989689-958d-4f13-8bb6-eff963fc2b72") : secret "maas-api-e2e-shared-a-4cd364-serving-cert" not found
Jul 24 14:13:59.762256 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.762238    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:13:59.773137 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:13:59.773108    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2s7n\" (UniqueName: \"kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:00.267838 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:00.267800    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:00.270559 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:00.270533    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:00.482607 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:00.482566    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:00.638397 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:00.638364    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:14:00.641138 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:14:00.641101    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84989689_958d_4f13_8bb6_eff963fc2b72.slice/crio-27cf5944ba071c2b3c7b0d446b8d8bb9bcb27fe0ba1958ec2c41e7f7030f26fc WatchSource:0}: Error finding container 27cf5944ba071c2b3c7b0d446b8d8bb9bcb27fe0ba1958ec2c41e7f7030f26fc: Status 404 returned error can't find the container with id 27cf5944ba071c2b3c7b0d446b8d8bb9bcb27fe0ba1958ec2c41e7f7030f26fc
Jul 24 14:14:01.479739 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:01.479703    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" event={"ID":"84989689-958d-4f13-8bb6-eff963fc2b72","Type":"ContainerStarted","Data":"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"}
Jul 24 14:14:01.479926 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:01.479764    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" event={"ID":"84989689-958d-4f13-8bb6-eff963fc2b72","Type":"ContainerStarted","Data":"27cf5944ba071c2b3c7b0d446b8d8bb9bcb27fe0ba1958ec2c41e7f7030f26fc"}
Jul 24 14:14:01.479926 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:01.479793    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:01.509868 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:01.509814    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" podStartSLOduration=2.087905864 podStartE2EDuration="2.509794829s" totalImagesPullingTime="421.888965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:01.507330823 +0000 UTC m=+2405.549378709" watchObservedRunningTime="2026-07-24 14:14:01.509794829 +0000 UTC m=+2405.551842713"
Jul 24 14:14:07.490317 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:07.490285    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:14:08.144822 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.144785    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:14:08.148464 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.148409    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.150763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.150740    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-525009-serving-cert\""
Jul 24 14:14:08.158664 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.158636    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:14:08.242894 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.242854    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.243089 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.242938    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.243089 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.242969    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv7n\" (UniqueName: \"kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.344075 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.344030    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.344262 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.344099    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.344262 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.344125    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv7n\" (UniqueName: \"kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.344262 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:14:08.344189    2527 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-serving-cert: secret "maas-api-e2e-shared-b-525009-serving-cert" not found
Jul 24 14:14:08.344262 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:14:08.344258    2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls podName:c89639ce-58c1-4afa-881f-4d44900403e0 nodeName:}" failed. No retries permitted until 2026-07-24 14:14:08.84424053 +0000 UTC m=+2412.886288393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls") pod "maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" (UID: "c89639ce-58c1-4afa-881f-4d44900403e0") : secret "maas-api-e2e-shared-b-525009-serving-cert" not found
Jul 24 14:14:08.344860 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.344840    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.354998 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.354968    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv7n\" (UniqueName: \"kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.848606 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.848568    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:08.851289 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:08.851265    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:09.061497 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.061462    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:09.222772 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.222731    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:14:09.226529 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:14:09.226496    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89639ce_58c1_4afa_881f_4d44900403e0.slice/crio-4adcf10126628f0ff0413a8f678e59ce5d8c66cb7234716b7c41855e90d65521 WatchSource:0}: Error finding container 4adcf10126628f0ff0413a8f678e59ce5d8c66cb7234716b7c41855e90d65521: Status 404 returned error can't find the container with id 4adcf10126628f0ff0413a8f678e59ce5d8c66cb7234716b7c41855e90d65521
Jul 24 14:14:09.290873 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.290838    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:14:09.295762 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.295738    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.299512 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.299485    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-525009\""
Jul 24 14:14:09.299708 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.299688    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-525009-dockercfg-k58wz\""
Jul 24 14:14:09.308142 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.308113    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:14:09.454953 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.454859    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26nvd\" (UniqueName: \"kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.454953 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.454921    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.520406 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.520373    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" event={"ID":"c89639ce-58c1-4afa-881f-4d44900403e0","Type":"ContainerStarted","Data":"4adcf10126628f0ff0413a8f678e59ce5d8c66cb7234716b7c41855e90d65521"}
Jul 24 14:14:09.555966 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.555923    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26nvd\" (UniqueName: \"kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.556169 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.555998    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.556825 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.556807    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.566454 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.566414    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26nvd\" (UniqueName: \"kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd\") pod \"payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") " pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.608725 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.608681    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:09.773061 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:09.773032    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:14:09.775517 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:14:09.775489    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59822856_46d8_4f9c_be49_d625f0318763.slice/crio-9847c3b1d64456aa4964c0161b107fe03ea7908bb2611e4ff445042b0030c98e WatchSource:0}: Error finding container 9847c3b1d64456aa4964c0161b107fe03ea7908bb2611e4ff445042b0030c98e: Status 404 returned error can't find the container with id 9847c3b1d64456aa4964c0161b107fe03ea7908bb2611e4ff445042b0030c98e
Jul 24 14:14:10.527275 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.527234    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" event={"ID":"59822856-46d8-4f9c-be49-d625f0318763","Type":"ContainerStarted","Data":"85a0f3765dcced03f264616276497c3558c2442f3ad83db5e0a53254daba788e"}
Jul 24 14:14:10.527808 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.527303    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" event={"ID":"59822856-46d8-4f9c-be49-d625f0318763","Type":"ContainerStarted","Data":"9847c3b1d64456aa4964c0161b107fe03ea7908bb2611e4ff445042b0030c98e"}
Jul 24 14:14:10.527808 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.527347    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:14:10.528945 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.528920    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" event={"ID":"c89639ce-58c1-4afa-881f-4d44900403e0","Type":"ContainerStarted","Data":"491f261b05aea4381ba2e78cea967dcc025e6dc45e12fd0b93843a53529233fe"}
Jul 24 14:14:10.529080 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.529057    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:10.546130 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.546074    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" podStartSLOduration=1.5460573210000002 podStartE2EDuration="1.546057321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:10.545243188 +0000 UTC m=+2414.587291107" watchObservedRunningTime="2026-07-24 14:14:10.546057321 +0000 UTC m=+2414.588105184"
Jul 24 14:14:10.564112 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:10.564045    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" podStartSLOduration=2.14222569 podStartE2EDuration="2.564024857s" totalImagesPullingTime="421.799167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:10.562559967 +0000 UTC m=+2414.604607851" watchObservedRunningTime="2026-07-24 14:14:10.564024857 +0000 UTC m=+2414.606072744"
Jul 24 14:14:16.539652 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:16.539623    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:14:21.537029 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:14:21.536956    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:15:00.139379 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.139341    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"]
Jul 24 14:15:00.148842 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.148806    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"
Jul 24 14:15:00.152200 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.152175    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:15:00.154464 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.154418    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"]
Jul 24 14:15:00.161381 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.161360    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"
Jul 24 14:15:00.311144 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.311118    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"]
Jul 24 14:15:00.314410 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:15:00.314378    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c7787b_43ee_4685_be2d_e11f42597842.slice/crio-f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276 WatchSource:0}: Error finding container f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276: Status 404 returned error can't find the container with id f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276
Jul 24 14:15:00.789897 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:00.789863    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f" event={"ID":"58c7787b-43ee-4685-be2d-e11f42597842","Type":"ContainerStarted","Data":"f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276"}
Jul 24 14:15:02.802564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:02.802524    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f" event={"ID":"58c7787b-43ee-4685-be2d-e11f42597842","Type":"ContainerStarted","Data":"2813d63a0d61286dc67c195c851e4bfcb55075990f4fde2c112d216d6235858c"}
Jul 24 14:15:02.819085 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:02.819033    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f" podStartSLOduration=0.827673493 podStartE2EDuration="2.819015208s" totalImagesPullingTime="1.991341715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:02.816990224 +0000 UTC m=+2466.859038109" watchObservedRunningTime="2026-07-24 14:15:02.819015208 +0000 UTC m=+2466.861063092"
Jul 24 14:15:03.809588 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:03.809551    2527 generic.go:356] "Generic (PLEG): container finished" podID="58c7787b-43ee-4685-be2d-e11f42597842" containerID="2813d63a0d61286dc67c195c851e4bfcb55075990f4fde2c112d216d6235858c" exitCode=0
Jul 24 14:15:03.809999 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:03.809626    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f" event={"ID":"58c7787b-43ee-4685-be2d-e11f42597842","Type":"ContainerDied","Data":"2813d63a0d61286dc67c195c851e4bfcb55075990f4fde2c112d216d6235858c"}
Jul 24 14:15:04.959005 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:04.958976    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"
Jul 24 14:15:05.820773 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:05.820740    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f" event={"ID":"58c7787b-43ee-4685-be2d-e11f42597842","Type":"ContainerDied","Data":"f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276"}
Jul 24 14:15:05.820773 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:05.820772    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"
Jul 24 14:15:05.821068 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:15:05.820788    2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f204b6189c504d8a0c81bcf4240e371653bf07f481e25da5d046ed095286d276"
Jul 24 14:18:56.601795 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:18:56.601711    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:18:56.603730 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:18:56.603708    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:20:23.676997 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.676890    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:20:23.677588 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.677369    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c7787b-43ee-4685-be2d-e11f42597842" containerName="cleanup"
Jul 24 14:20:23.677588 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.677385    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c7787b-43ee-4685-be2d-e11f42597842" containerName="cleanup"
Jul 24 14:20:23.677588 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.677492    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c7787b-43ee-4685-be2d-e11f42597842" containerName="cleanup"
Jul 24 14:20:23.680809 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.680791    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.682837 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.682813    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-546fcb-serving-cert\""
Jul 24 14:20:23.696679 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.696649    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:20:23.732259 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.732226    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.732506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.732284    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdm9\" (UniqueName: \"kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.732506 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.732337    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.833125 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.833085    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdm9\" (UniqueName: \"kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.833125 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.833129    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.833353 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.833198    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.834095 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.834018    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.836010 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.835986    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.850235 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.850206    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdm9\" (UniqueName: \"kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9\") pod \"maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:23.993760 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:23.993669    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:24.140731 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:24.140703    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:20:24.143341 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:20:24.143309    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7239288_2b0d_4da9_8638_d079df45e099.slice/crio-e10157e74cb9d6ffc164b1473763b4480320fa2814d02fbff627ea5b92646629 WatchSource:0}: Error finding container e10157e74cb9d6ffc164b1473763b4480320fa2814d02fbff627ea5b92646629: Status 404 returned error can't find the container with id e10157e74cb9d6ffc164b1473763b4480320fa2814d02fbff627ea5b92646629
Jul 24 14:20:24.144869 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:24.144849    2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:20:24.465938 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:24.465896    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" event={"ID":"e7239288-2b0d-4da9-8638-d079df45e099","Type":"ContainerStarted","Data":"e10157e74cb9d6ffc164b1473763b4480320fa2814d02fbff627ea5b92646629"}
Jul 24 14:20:25.472783 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:25.472746    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" event={"ID":"e7239288-2b0d-4da9-8638-d079df45e099","Type":"ContainerStarted","Data":"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"}
Jul 24 14:20:25.473193 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:25.472821    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:25.511134 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:25.511081    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" podStartSLOduration=1.836520743 podStartE2EDuration="2.511064214s" totalImagesPullingTime="674.543471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:25.509867772 +0000 UTC m=+2789.551915656" watchObservedRunningTime="2026-07-24 14:20:25.511064214 +0000 UTC m=+2789.553112145"
Jul 24 14:20:31.483105 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:31.483072    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:20:48.626587 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.626551    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:20:48.630481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.630419    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.636488 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.636460    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-359bb9-serving-cert\""
Jul 24 14:20:48.658959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.658923    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:20:48.768197 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.768153    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.768412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.768209    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58ts\" (UniqueName: \"kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.768412 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.768334    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.869110 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.869075    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.869110 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.869113    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.869319 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.869142    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58ts\" (UniqueName: \"kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.869931 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.869868    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.871799 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.871776    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.886112 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.886044    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58ts\" (UniqueName: \"kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts\") pod \"maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:48.945526 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:48.945495    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:49.103387 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.103349    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:20:49.107012 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:20:49.106981    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf81f42_6a1e_4371_8b33_7e138fce8cb0.slice/crio-913b3b85303ccc1bee21778dd4a204b1a1e4ac4d5d71a106a2d0ca87ae10e71c WatchSource:0}: Error finding container 913b3b85303ccc1bee21778dd4a204b1a1e4ac4d5d71a106a2d0ca87ae10e71c: Status 404 returned error can't find the container with id 913b3b85303ccc1bee21778dd4a204b1a1e4ac4d5d71a106a2d0ca87ae10e71c
Jul 24 14:20:49.605700 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.605665    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" event={"ID":"2cf81f42-6a1e-4371-8b33-7e138fce8cb0","Type":"ContainerStarted","Data":"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"}
Jul 24 14:20:49.605903 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.605722    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" event={"ID":"2cf81f42-6a1e-4371-8b33-7e138fce8cb0","Type":"ContainerStarted","Data":"913b3b85303ccc1bee21778dd4a204b1a1e4ac4d5d71a106a2d0ca87ae10e71c"}
Jul 24 14:20:49.605903 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.605800    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:20:49.634982 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.634926    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" podStartSLOduration=1.223704122 podStartE2EDuration="1.634908591s" totalImagesPullingTime="411.204469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:49.63112897 +0000 UTC m=+2813.673176852" watchObservedRunningTime="2026-07-24 14:20:49.634908591 +0000 UTC m=+2813.676956477"
Jul 24 14:20:49.851683 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.851646    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:20:49.855522 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.855496    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:49.858554 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.858527    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-359bb9-dockercfg-4pt56\""
Jul 24 14:20:49.858877 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.858856    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-359bb9\""
Jul 24 14:20:49.873958 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.873929    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:20:49.979945 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.979907    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5zb\" (UniqueName: \"kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:49.980121 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:49.979975    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.080531 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.080493    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5zb\" (UniqueName: \"kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.080714 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.080571    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.081248 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.081229    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.097961 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.097932    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5zb\" (UniqueName: \"kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb\") pod \"payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.166289 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.166194    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:50.327360 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.327332    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:20:50.329119 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:20:50.329094    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76df9c1a_5e6a_4ea5_a28a_8f2bdb6a0b48.slice/crio-d0b93d987362edb96a15dddabd6bd2f664b51993df5c56d4305459526c12b0ad WatchSource:0}: Error finding container d0b93d987362edb96a15dddabd6bd2f664b51993df5c56d4305459526c12b0ad: Status 404 returned error can't find the container with id d0b93d987362edb96a15dddabd6bd2f664b51993df5c56d4305459526c12b0ad
Jul 24 14:20:50.619989 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.619950    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" event={"ID":"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48","Type":"ContainerStarted","Data":"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"}
Jul 24 14:20:50.620187 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.620041    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" event={"ID":"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48","Type":"ContainerStarted","Data":"d0b93d987362edb96a15dddabd6bd2f664b51993df5c56d4305459526c12b0ad"}
Jul 24 14:20:50.640523 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:50.640469    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" podStartSLOduration=1.6404233879999999 podStartE2EDuration="1.640423388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:50.638445068 +0000 UTC m=+2814.680492946" watchObservedRunningTime="2026-07-24 14:20:50.640423388 +0000 UTC m=+2814.682471275"
Jul 24 14:20:51.626334 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:51.626294    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:20:55.626160 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:20:55.626129    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:21:02.635937 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:02.635900    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:21:12.854365 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:12.854329    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:21:12.984806 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:12.984771    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp_76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48/payload-processing/0.log"
Jul 24 14:21:52.678065 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:52.677979    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:21:52.816003 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:52.815976    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp_76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48/payload-processing/0.log"
Jul 24 14:21:59.378531 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.378489    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:21:59.382363 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.382338    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.384743 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.384720    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-269297-serving-cert\""
Jul 24 14:21:59.400663 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.400632    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:21:59.428713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.428667    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.428713 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.428708    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.429009 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.428785    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgb5\" (UniqueName: \"kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.529686 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.529650    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.529686 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.529691    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.529919 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.529734    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgb5\" (UniqueName: \"kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.530349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.530318    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.532336 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.532313    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.542407 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.542375    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgb5\" (UniqueName: \"kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5\") pod \"maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.693719 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.693634    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:21:59.853395 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.853369    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:21:59.855809 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:21:59.855775    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396515dd_9e64_40b9_95cd_656caa3bfb0b.slice/crio-8208bc3ac5a36ab6f234755e21b22ddd6c59b67f23208ff01034bfe2914fb04a WatchSource:0}: Error finding container 8208bc3ac5a36ab6f234755e21b22ddd6c59b67f23208ff01034bfe2914fb04a: Status 404 returned error can't find the container with id 8208bc3ac5a36ab6f234755e21b22ddd6c59b67f23208ff01034bfe2914fb04a
Jul 24 14:21:59.987135 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:21:59.987045    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" event={"ID":"396515dd-9e64-40b9-95cd-656caa3bfb0b","Type":"ContainerStarted","Data":"8208bc3ac5a36ab6f234755e21b22ddd6c59b67f23208ff01034bfe2914fb04a"}
Jul 24 14:22:00.993808 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:00.993770    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" event={"ID":"396515dd-9e64-40b9-95cd-656caa3bfb0b","Type":"ContainerStarted","Data":"48f2d0b255f1ad1b68e461b06d94dfe651b25ed5fc9205a91312c0ad977a1f40"}
Jul 24 14:22:00.994273 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:00.993855    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:22:01.014782 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:01.014733    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" podStartSLOduration=1.5749520700000001 podStartE2EDuration="2.014715032s" totalImagesPullingTime="439.762962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:22:01.01451979 +0000 UTC m=+2885.056567675" watchObservedRunningTime="2026-07-24 14:22:01.014715032 +0000 UTC m=+2885.056762916"
Jul 24 14:22:07.004561 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:07.004525    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:22:30.946589 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:30.946552    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:22:30.949358 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:30.946801    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" podUID="396515dd-9e64-40b9-95cd-656caa3bfb0b" containerName="maas-api" containerID="cri-o://48f2d0b255f1ad1b68e461b06d94dfe651b25ed5fc9205a91312c0ad977a1f40" gracePeriod=30
Jul 24 14:22:31.156542 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.156508    2527 generic.go:356] "Generic (PLEG): container finished" podID="396515dd-9e64-40b9-95cd-656caa3bfb0b" containerID="48f2d0b255f1ad1b68e461b06d94dfe651b25ed5fc9205a91312c0ad977a1f40" exitCode=0
Jul 24 14:22:31.156731 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.156564    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" event={"ID":"396515dd-9e64-40b9-95cd-656caa3bfb0b","Type":"ContainerDied","Data":"48f2d0b255f1ad1b68e461b06d94dfe651b25ed5fc9205a91312c0ad977a1f40"}
Jul 24 14:22:31.212043 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.211973    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:22:31.224224 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.224194    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca\") pod \"396515dd-9e64-40b9-95cd-656caa3bfb0b\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") "
Jul 24 14:22:31.224372 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.224294    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls\") pod \"396515dd-9e64-40b9-95cd-656caa3bfb0b\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") "
Jul 24 14:22:31.224419 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.224389    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgb5\" (UniqueName: \"kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5\") pod \"396515dd-9e64-40b9-95cd-656caa3bfb0b\" (UID: \"396515dd-9e64-40b9-95cd-656caa3bfb0b\") "
Jul 24 14:22:31.224660 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.224629    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "396515dd-9e64-40b9-95cd-656caa3bfb0b" (UID: "396515dd-9e64-40b9-95cd-656caa3bfb0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:22:31.224857 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.224839    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396515dd-9e64-40b9-95cd-656caa3bfb0b-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.226601 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.226576    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "396515dd-9e64-40b9-95cd-656caa3bfb0b" (UID: "396515dd-9e64-40b9-95cd-656caa3bfb0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:22:31.226601 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.226584    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5" (OuterVolumeSpecName: "kube-api-access-8wgb5") pod "396515dd-9e64-40b9-95cd-656caa3bfb0b" (UID: "396515dd-9e64-40b9-95cd-656caa3bfb0b"). InnerVolumeSpecName "kube-api-access-8wgb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:22:31.325698 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.325656    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396515dd-9e64-40b9-95cd-656caa3bfb0b-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.325698 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:31.325690    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgb5\" (UniqueName: \"kubernetes.io/projected/396515dd-9e64-40b9-95cd-656caa3bfb0b-kube-api-access-8wgb5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:22:32.162893 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.162862    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp" event={"ID":"396515dd-9e64-40b9-95cd-656caa3bfb0b","Type":"ContainerDied","Data":"8208bc3ac5a36ab6f234755e21b22ddd6c59b67f23208ff01034bfe2914fb04a"}
Jul 24 14:22:32.162893 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.162886    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"
Jul 24 14:22:32.163392 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.162918    2527 scope.go:117] "RemoveContainer" containerID="48f2d0b255f1ad1b68e461b06d94dfe651b25ed5fc9205a91312c0ad977a1f40"
Jul 24 14:22:32.186070 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.186041    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:22:32.188758 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.188731    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-269297-b55d7ddf6-6p7dp"]
Jul 24 14:22:32.521243 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:32.521159    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396515dd-9e64-40b9-95cd-656caa3bfb0b" path="/var/lib/kubelet/pods/396515dd-9e64-40b9-95cd-656caa3bfb0b/volumes"
Jul 24 14:22:44.007282 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.007247    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"]
Jul 24 14:22:44.007763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.007715    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396515dd-9e64-40b9-95cd-656caa3bfb0b" containerName="maas-api"
Jul 24 14:22:44.007763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.007727    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="396515dd-9e64-40b9-95cd-656caa3bfb0b" containerName="maas-api"
Jul 24 14:22:44.007836 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.007791    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="396515dd-9e64-40b9-95cd-656caa3bfb0b" containerName="maas-api"
Jul 24 14:22:44.011580 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.011551    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:44.014201 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.014179    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:22:44.024418 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.024386    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"]
Jul 24 14:22:44.150246 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.150198    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf\" (UID: \"bdc7b5c5-2961-4e21-a70f-a3cf98857a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:44.250759 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.250719    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf\" (UID: \"bdc7b5c5-2961-4e21-a70f-a3cf98857a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:44.251390 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.251370    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf\" (UID: \"bdc7b5c5-2961-4e21-a70f-a3cf98857a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:44.321875 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.321787    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:44.469375 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:44.469338    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"]
Jul 24 14:22:44.472766 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:22:44.472729    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc7b5c5_2961_4e21_a70f_a3cf98857a56.slice/crio-745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a WatchSource:0}: Error finding container 745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a: Status 404 returned error can't find the container with id 745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a
Jul 24 14:22:45.241533 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:45.241496    2527 generic.go:356] "Generic (PLEG): container finished" podID="bdc7b5c5-2961-4e21-a70f-a3cf98857a56" containerID="e46ede5bfb18d9fab904dba9fd0d36c1123ea5c67025042d83d9dc929735cea0" exitCode=0
Jul 24 14:22:45.241952 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:45.241584    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf" event={"ID":"bdc7b5c5-2961-4e21-a70f-a3cf98857a56","Type":"ContainerDied","Data":"e46ede5bfb18d9fab904dba9fd0d36c1123ea5c67025042d83d9dc929735cea0"}
Jul 24 14:22:45.241952 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:45.241689    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf" event={"ID":"bdc7b5c5-2961-4e21-a70f-a3cf98857a56","Type":"ContainerStarted","Data":"745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a"}
Jul 24 14:22:46.396152 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:46.396127    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:46.472892 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:46.472852    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca\") pod \"bdc7b5c5-2961-4e21-a70f-a3cf98857a56\" (UID: \"bdc7b5c5-2961-4e21-a70f-a3cf98857a56\") "
Jul 24 14:22:46.473248 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:46.473221    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdc7b5c5-2961-4e21-a70f-a3cf98857a56" (UID: "bdc7b5c5-2961-4e21-a70f-a3cf98857a56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:22:46.574930 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:46.574893    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc7b5c5-2961-4e21-a70f-a3cf98857a56-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:22:47.255852 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:47.255812    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf" event={"ID":"bdc7b5c5-2961-4e21-a70f-a3cf98857a56","Type":"ContainerDied","Data":"745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a"}
Jul 24 14:22:47.255852 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:47.255842    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"
Jul 24 14:22:47.256077 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:22:47.255874    2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745e8561d37ed7b6e3ad4b8a036d5a2935085224bd885cfceb96d8680c8dc21a"
Jul 24 14:23:04.364257 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.364217    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:23:04.364813 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.364489    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" podUID="e7239288-2b0d-4da9-8638-d079df45e099" containerName="maas-api" containerID="cri-o://4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642" gracePeriod=30
Jul 24 14:23:04.633960 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.633928    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:23:04.748649 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.748614    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca\") pod \"e7239288-2b0d-4da9-8638-d079df45e099\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") "
Jul 24 14:23:04.748853 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.748703    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls\") pod \"e7239288-2b0d-4da9-8638-d079df45e099\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") "
Jul 24 14:23:04.748853 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.748771    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcdm9\" (UniqueName: \"kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9\") pod \"e7239288-2b0d-4da9-8638-d079df45e099\" (UID: \"e7239288-2b0d-4da9-8638-d079df45e099\") "
Jul 24 14:23:04.749044 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.749011    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7239288-2b0d-4da9-8638-d079df45e099" (UID: "e7239288-2b0d-4da9-8638-d079df45e099"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:23:04.750997 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.750967    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7239288-2b0d-4da9-8638-d079df45e099" (UID: "e7239288-2b0d-4da9-8638-d079df45e099"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:23:04.751143 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.751116    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9" (OuterVolumeSpecName: "kube-api-access-hcdm9") pod "e7239288-2b0d-4da9-8638-d079df45e099" (UID: "e7239288-2b0d-4da9-8638-d079df45e099"). InnerVolumeSpecName "kube-api-access-hcdm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:23:04.850367 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.850333    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7239288-2b0d-4da9-8638-d079df45e099-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:23:04.850367 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.850362    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7239288-2b0d-4da9-8638-d079df45e099-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:23:04.850367 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:04.850372    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdm9\" (UniqueName: \"kubernetes.io/projected/e7239288-2b0d-4da9-8638-d079df45e099-kube-api-access-hcdm9\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:23:05.356041 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.356004    2527 generic.go:356] "Generic (PLEG): container finished" podID="e7239288-2b0d-4da9-8638-d079df45e099" containerID="4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642" exitCode=0
Jul 24 14:23:05.356198 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.356045    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"
Jul 24 14:23:05.356198 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.356098    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" event={"ID":"e7239288-2b0d-4da9-8638-d079df45e099","Type":"ContainerDied","Data":"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"}
Jul 24 14:23:05.356198 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.356154    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4" event={"ID":"e7239288-2b0d-4da9-8638-d079df45e099","Type":"ContainerDied","Data":"e10157e74cb9d6ffc164b1473763b4480320fa2814d02fbff627ea5b92646629"}
Jul 24 14:23:05.356327 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.356209    2527 scope.go:117] "RemoveContainer" containerID="4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"
Jul 24 14:23:05.366584 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.366375    2527 scope.go:117] "RemoveContainer" containerID="4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"
Jul 24 14:23:05.366861 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:23:05.366747    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642\": container with ID starting with 4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642 not found: ID does not exist" containerID="4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"
Jul 24 14:23:05.366861 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.366775    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642"} err="failed to get container status \"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642\": rpc error: code = NotFound desc = could not find container \"4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642\": container with ID starting with 4c2cd23e478d844b39a510c9205e9a39acb548fb7351ec25a70a5902ff05c642 not found: ID does not exist"
Jul 24 14:23:05.376883 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.376855    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:23:05.382420 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:05.382394    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-546fcb-79994946f8-ltnr4"]
Jul 24 14:23:06.514655 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:06.514624    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7239288-2b0d-4da9-8638-d079df45e099" path="/var/lib/kubelet/pods/e7239288-2b0d-4da9-8638-d079df45e099/volumes"
Jul 24 14:23:09.153539 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:09.153499    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"]
Jul 24 14:23:09.158210 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:09.158185    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-546fcb-6dnkf"]
Jul 24 14:23:10.514066 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:10.514034    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc7b5c5-2961-4e21-a70f-a3cf98857a56" path="/var/lib/kubelet/pods/bdc7b5c5-2961-4e21-a70f-a3cf98857a56/volumes"
Jul 24 14:23:56.638988 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:56.638950    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:23:56.640553 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:23:56.640520    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:24:11.696896 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:11.696862    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:24:11.697741 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:11.697099    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" podUID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" containerName="maas-api" containerID="cri-o://567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80" gracePeriod=30
Jul 24 14:24:11.778862 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:11.778826    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:24:11.779150 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:11.779119    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" podUID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" containerName="payload-processing" containerID="cri-o://62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c" gracePeriod=30
Jul 24 14:24:12.048301 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.048275    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:24:12.074016 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.073990    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:24:12.180589 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.180550    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw5zb\" (UniqueName: \"kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb\") pod \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") "
Jul 24 14:24:12.180763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.180606    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls\") pod \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") "
Jul 24 14:24:12.180763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.180688    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w58ts\" (UniqueName: \"kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts\") pod \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") "
Jul 24 14:24:12.180763 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.180738    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume\") pod \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\" (UID: \"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48\") "
Jul 24 14:24:12.180893 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.180773    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca\") pod \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\" (UID: \"2cf81f42-6a1e-4371-8b33-7e138fce8cb0\") "
Jul 24 14:24:12.181174 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.181142    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" (UID: "76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:12.181174 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.181163    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2cf81f42-6a1e-4371-8b33-7e138fce8cb0" (UID: "2cf81f42-6a1e-4371-8b33-7e138fce8cb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:12.183013 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.182991    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb" (OuterVolumeSpecName: "kube-api-access-jw5zb") pod "76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" (UID: "76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48"). InnerVolumeSpecName "kube-api-access-jw5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:12.183117 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.183016    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2cf81f42-6a1e-4371-8b33-7e138fce8cb0" (UID: "2cf81f42-6a1e-4371-8b33-7e138fce8cb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:24:12.183117 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.183073    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts" (OuterVolumeSpecName: "kube-api-access-w58ts") pod "2cf81f42-6a1e-4371-8b33-7e138fce8cb0" (UID: "2cf81f42-6a1e-4371-8b33-7e138fce8cb0"). InnerVolumeSpecName "kube-api-access-w58ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:12.281994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.281908    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.281994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.281939    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5zb\" (UniqueName: \"kubernetes.io/projected/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-kube-api-access-jw5zb\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.281994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.281949    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.281994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.281959    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58ts\" (UniqueName: \"kubernetes.io/projected/2cf81f42-6a1e-4371-8b33-7e138fce8cb0-kube-api-access-w58ts\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.281994 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.281967    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.712790 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.712678    2527 generic.go:356] "Generic (PLEG): container finished" podID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" containerID="567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80" exitCode=0
Jul 24 14:24:12.712790 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.712750    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"
Jul 24 14:24:12.712790 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.712756    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" event={"ID":"2cf81f42-6a1e-4371-8b33-7e138fce8cb0","Type":"ContainerDied","Data":"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"}
Jul 24 14:24:12.713242 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.712822    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn" event={"ID":"2cf81f42-6a1e-4371-8b33-7e138fce8cb0","Type":"ContainerDied","Data":"913b3b85303ccc1bee21778dd4a204b1a1e4ac4d5d71a106a2d0ca87ae10e71c"}
Jul 24 14:24:12.713793 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.713075    2527 scope.go:117] "RemoveContainer" containerID="567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"
Jul 24 14:24:12.715293 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.715103    2527 generic.go:356] "Generic (PLEG): container finished" podID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" containerID="62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c" exitCode=0
Jul 24 14:24:12.715293 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.715165    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"
Jul 24 14:24:12.715293 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.715201    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" event={"ID":"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48","Type":"ContainerDied","Data":"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"}
Jul 24 14:24:12.715293 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.715291    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp" event={"ID":"76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48","Type":"ContainerDied","Data":"d0b93d987362edb96a15dddabd6bd2f664b51993df5c56d4305459526c12b0ad"}
Jul 24 14:24:12.723705 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.723684    2527 scope.go:117] "RemoveContainer" containerID="567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"
Jul 24 14:24:12.724010 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:24:12.723989    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80\": container with ID starting with 567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80 not found: ID does not exist" containerID="567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"
Jul 24 14:24:12.724058 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.724019    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80"} err="failed to get container status \"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80\": rpc error: code = NotFound desc = could not find container \"567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80\": container with ID starting with 567ba54ed7fc8cf2dbcb8e00d9ad4ad987b2c7d1a5ffba30f04f245826089b80 not found: ID does not exist"
Jul 24 14:24:12.724058 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.724037    2527 scope.go:117] "RemoveContainer" containerID="62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"
Jul 24 14:24:12.732948 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.732921    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:24:12.733239 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.733218    2527 scope.go:117] "RemoveContainer" containerID="62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"
Jul 24 14:24:12.733718 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:24:12.733668    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c\": container with ID starting with 62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c not found: ID does not exist" containerID="62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"
Jul 24 14:24:12.733718 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.733704    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c"} err="failed to get container status \"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c\": rpc error: code = NotFound desc = could not find container \"62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c\": container with ID starting with 62bccf4f54c3e3b1749083b742f799cd75cb705e86b6c87dadfc200c6402099c not found: ID does not exist"
Jul 24 14:24:12.735191 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.735173    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-359bb9-6c7b4655b8-h89hp"]
Jul 24 14:24:12.746550 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.746516    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:24:12.748678 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:12.748653    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-359bb9-788dd7dd79-pvckn"]
Jul 24 14:24:14.513987 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:14.513955    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" path="/var/lib/kubelet/pods/2cf81f42-6a1e-4371-8b33-7e138fce8cb0/volumes"
Jul 24 14:24:14.514474 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:14.514365    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" path="/var/lib/kubelet/pods/76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48/volumes"
Jul 24 14:24:23.068334 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068294    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"]
Jul 24 14:24:23.068759 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068744    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7239288-2b0d-4da9-8638-d079df45e099" containerName="maas-api"
Jul 24 14:24:23.068802 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068761    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7239288-2b0d-4da9-8638-d079df45e099" containerName="maas-api"
Jul 24 14:24:23.068802 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068777    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc7b5c5-2961-4e21-a70f-a3cf98857a56" containerName="revoke-keys"
Jul 24 14:24:23.068802 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068783    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc7b5c5-2961-4e21-a70f-a3cf98857a56" containerName="revoke-keys"
Jul 24 14:24:23.068802 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068790    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" containerName="payload-processing"
Jul 24 14:24:23.068802 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068797    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" containerName="payload-processing"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068819    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" containerName="maas-api"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068824    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" containerName="maas-api"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068880    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf81f42-6a1e-4371-8b33-7e138fce8cb0" containerName="maas-api"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068888    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7239288-2b0d-4da9-8638-d079df45e099" containerName="maas-api"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068895    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="76df9c1a-5e6a-4ea5-a28a-8f2bdb6a0b48" containerName="payload-processing"
Jul 24 14:24:23.068959 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.068901    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc7b5c5-2961-4e21-a70f-a3cf98857a56" containerName="revoke-keys"
Jul 24 14:24:23.073674 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.073653    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:23.076054 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.076030    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:24:23.085258 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.085229    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"]
Jul 24 14:24:23.184990 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.184941    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw\" (UID: \"9ffd4034-f014-4133-a727-f4f361eba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:23.286557 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.286521    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw\" (UID: \"9ffd4034-f014-4133-a727-f4f361eba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:23.287162 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.287128    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw\" (UID: \"9ffd4034-f014-4133-a727-f4f361eba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:23.385896 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.385846    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:23.551380 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.551351    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"]
Jul 24 14:24:23.553541 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:24:23.553499    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffd4034_f014_4133_a727_f4f361eba929.slice/crio-7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6 WatchSource:0}: Error finding container 7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6: Status 404 returned error can't find the container with id 7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6
Jul 24 14:24:23.775736 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:23.775642    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw" event={"ID":"9ffd4034-f014-4133-a727-f4f361eba929","Type":"ContainerStarted","Data":"7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6"}
Jul 24 14:24:24.782060 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:24.782024    2527 generic.go:356] "Generic (PLEG): container finished" podID="9ffd4034-f014-4133-a727-f4f361eba929" containerID="d91d6d54a52f81053f2f74786cd88478bc319359570dc81015b1b6ccf8cf7896" exitCode=0
Jul 24 14:24:24.782488 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:24.782107    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw" event={"ID":"9ffd4034-f014-4133-a727-f4f361eba929","Type":"ContainerDied","Data":"d91d6d54a52f81053f2f74786cd88478bc319359570dc81015b1b6ccf8cf7896"}
Jul 24 14:24:25.934985 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:25.934960    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:26.010755 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.010721    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca\") pod \"9ffd4034-f014-4133-a727-f4f361eba929\" (UID: \"9ffd4034-f014-4133-a727-f4f361eba929\") "
Jul 24 14:24:26.011069 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.011048    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ffd4034-f014-4133-a727-f4f361eba929" (UID: "9ffd4034-f014-4133-a727-f4f361eba929"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:26.112381 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.112348    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffd4034-f014-4133-a727-f4f361eba929-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:26.794709 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.794621    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"
Jul 24 14:24:26.794879 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.794625    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw" event={"ID":"9ffd4034-f014-4133-a727-f4f361eba929","Type":"ContainerDied","Data":"7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6"}
Jul 24 14:24:26.794879 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:26.794742    2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b646bbb7a999c15ea12af9a873ecd4b8ccfff56c14093a5818c780b4515c5e6"
Jul 24 14:24:33.983381 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:33.983327    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:24:33.983905 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:33.983585    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" podUID="84989689-958d-4f13-8bb6-eff963fc2b72" containerName="maas-api" containerID="cri-o://3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f" gracePeriod=30
Jul 24 14:24:34.243907 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.243826    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:24:34.389163 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.389125    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca\") pod \"84989689-958d-4f13-8bb6-eff963fc2b72\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") "
Jul 24 14:24:34.389356 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.389211    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") pod \"84989689-958d-4f13-8bb6-eff963fc2b72\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") "
Jul 24 14:24:34.389356 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.389252    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2s7n\" (UniqueName: \"kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n\") pod \"84989689-958d-4f13-8bb6-eff963fc2b72\" (UID: \"84989689-958d-4f13-8bb6-eff963fc2b72\") "
Jul 24 14:24:34.389589 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.389561    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84989689-958d-4f13-8bb6-eff963fc2b72" (UID: "84989689-958d-4f13-8bb6-eff963fc2b72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:34.391662 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.391627    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84989689-958d-4f13-8bb6-eff963fc2b72" (UID: "84989689-958d-4f13-8bb6-eff963fc2b72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:24:34.391770 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.391693    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n" (OuterVolumeSpecName: "kube-api-access-l2s7n") pod "84989689-958d-4f13-8bb6-eff963fc2b72" (UID: "84989689-958d-4f13-8bb6-eff963fc2b72"). InnerVolumeSpecName "kube-api-access-l2s7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:34.490131 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.490096    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2s7n\" (UniqueName: \"kubernetes.io/projected/84989689-958d-4f13-8bb6-eff963fc2b72-kube-api-access-l2s7n\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.490131 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.490126    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84989689-958d-4f13-8bb6-eff963fc2b72-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.490131 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.490136    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84989689-958d-4f13-8bb6-eff963fc2b72-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.835961 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.835926    2527 generic.go:356] "Generic (PLEG): container finished" podID="84989689-958d-4f13-8bb6-eff963fc2b72" containerID="3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f" exitCode=0
Jul 24 14:24:34.836168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.835988    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"
Jul 24 14:24:34.836168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.836010    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" event={"ID":"84989689-958d-4f13-8bb6-eff963fc2b72","Type":"ContainerDied","Data":"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"}
Jul 24 14:24:34.836168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.836069    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx" event={"ID":"84989689-958d-4f13-8bb6-eff963fc2b72","Type":"ContainerDied","Data":"27cf5944ba071c2b3c7b0d446b8d8bb9bcb27fe0ba1958ec2c41e7f7030f26fc"}
Jul 24 14:24:34.836168 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.836092    2527 scope.go:117] "RemoveContainer" containerID="3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"
Jul 24 14:24:34.845706 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.845686    2527 scope.go:117] "RemoveContainer" containerID="3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"
Jul 24 14:24:34.846028 ip-10-0-129-199 kubenswrapper[2527]: E0724 14:24:34.846003    2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f\": container with ID starting with 3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f not found: ID does not exist" containerID="3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"
Jul 24 14:24:34.846077 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.846037    2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f"} err="failed to get container status \"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f\": rpc error: code = NotFound desc = could not find container \"3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f\": container with ID starting with 3a8067c0fe6482c55dc58cbb6d023185a7a968f748da3dce61751d32ad207a2f not found: ID does not exist"
Jul 24 14:24:34.857345 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.857318    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:24:34.863855 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:34.863832    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cd364-6854cb5574-z27fx"]
Jul 24 14:24:36.515860 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:36.515827    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84989689-958d-4f13-8bb6-eff963fc2b72" path="/var/lib/kubelet/pods/84989689-958d-4f13-8bb6-eff963fc2b72/volumes"
Jul 24 14:24:38.209113 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:38.209076    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"]
Jul 24 14:24:38.211633 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:38.211603    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cd364-v28vw"]
Jul 24 14:24:38.514755 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:38.514670    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffd4034-f014-4133-a727-f4f361eba929" path="/var/lib/kubelet/pods/9ffd4034-f014-4133-a727-f4f361eba929/volumes"
Jul 24 14:24:57.657040 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.656994    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:24:57.657577 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.657266    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" podUID="c89639ce-58c1-4afa-881f-4d44900403e0" containerName="maas-api" containerID="cri-o://491f261b05aea4381ba2e78cea967dcc025e6dc45e12fd0b93843a53529233fe" gracePeriod=30
Jul 24 14:24:57.678474 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.678420    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"]
Jul 24 14:24:57.687124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.687085    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-525009-29748375-4pg9f"]
Jul 24 14:24:57.740313 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.740273    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:24:57.740645 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.740585    2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" podUID="59822856-46d8-4f9c-be49-d625f0318763" containerName="payload-processing" containerID="cri-o://85a0f3765dcced03f264616276497c3558c2442f3ad83db5e0a53254daba788e" gracePeriod=30
Jul 24 14:24:57.963009 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.962972    2527 generic.go:356] "Generic (PLEG): container finished" podID="c89639ce-58c1-4afa-881f-4d44900403e0" containerID="491f261b05aea4381ba2e78cea967dcc025e6dc45e12fd0b93843a53529233fe" exitCode=0
Jul 24 14:24:57.963177 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.963046    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" event={"ID":"c89639ce-58c1-4afa-881f-4d44900403e0","Type":"ContainerDied","Data":"491f261b05aea4381ba2e78cea967dcc025e6dc45e12fd0b93843a53529233fe"}
Jul 24 14:24:57.965145 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.965115    2527 generic.go:356] "Generic (PLEG): container finished" podID="59822856-46d8-4f9c-be49-d625f0318763" containerID="85a0f3765dcced03f264616276497c3558c2442f3ad83db5e0a53254daba788e" exitCode=0
Jul 24 14:24:57.965263 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:57.965187    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" event={"ID":"59822856-46d8-4f9c-be49-d625f0318763","Type":"ContainerDied","Data":"85a0f3765dcced03f264616276497c3558c2442f3ad83db5e0a53254daba788e"}
Jul 24 14:24:58.005602 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.005520    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:24:58.037489 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.037462    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:24:58.099599 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.099555    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") pod \"c89639ce-58c1-4afa-881f-4d44900403e0\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") "
Jul 24 14:24:58.099820 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.099648    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26nvd\" (UniqueName: \"kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd\") pod \"59822856-46d8-4f9c-be49-d625f0318763\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") "
Jul 24 14:24:58.099820 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.099707    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume\") pod \"59822856-46d8-4f9c-be49-d625f0318763\" (UID: \"59822856-46d8-4f9c-be49-d625f0318763\") "
Jul 24 14:24:58.099820 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.099763    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lv7n\" (UniqueName: \"kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n\") pod \"c89639ce-58c1-4afa-881f-4d44900403e0\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") "
Jul 24 14:24:58.099820 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.099790    2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca\") pod \"c89639ce-58c1-4afa-881f-4d44900403e0\" (UID: \"c89639ce-58c1-4afa-881f-4d44900403e0\") "
Jul 24 14:24:58.100186 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.100153    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c89639ce-58c1-4afa-881f-4d44900403e0" (UID: "c89639ce-58c1-4afa-881f-4d44900403e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:58.100186 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.100164    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59822856-46d8-4f9c-be49-d625f0318763" (UID: "59822856-46d8-4f9c-be49-d625f0318763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:58.101917 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.101892    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c89639ce-58c1-4afa-881f-4d44900403e0" (UID: "c89639ce-58c1-4afa-881f-4d44900403e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:24:58.102024 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.101969    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd" (OuterVolumeSpecName: "kube-api-access-26nvd") pod "59822856-46d8-4f9c-be49-d625f0318763" (UID: "59822856-46d8-4f9c-be49-d625f0318763"). InnerVolumeSpecName "kube-api-access-26nvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:58.102024 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.101999    2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n" (OuterVolumeSpecName: "kube-api-access-7lv7n") pod "c89639ce-58c1-4afa-881f-4d44900403e0" (UID: "c89639ce-58c1-4afa-881f-4d44900403e0"). InnerVolumeSpecName "kube-api-access-7lv7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:58.201148 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.201061    2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59822856-46d8-4f9c-be49-d625f0318763-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.201148 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.201091    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lv7n\" (UniqueName: \"kubernetes.io/projected/c89639ce-58c1-4afa-881f-4d44900403e0-kube-api-access-7lv7n\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.201148 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.201103    2527 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89639ce-58c1-4afa-881f-4d44900403e0-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.201148 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.201112    2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89639ce-58c1-4afa-881f-4d44900403e0-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.201148 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.201123    2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26nvd\" (UniqueName: \"kubernetes.io/projected/59822856-46d8-4f9c-be49-d625f0318763-kube-api-access-26nvd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.514728 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.514652    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c7787b-43ee-4685-be2d-e11f42597842" path="/var/lib/kubelet/pods/58c7787b-43ee-4685-be2d-e11f42597842/volumes"
Jul 24 14:24:58.971651 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.971623    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"
Jul 24 14:24:58.972110 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.971676    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m" event={"ID":"c89639ce-58c1-4afa-881f-4d44900403e0","Type":"ContainerDied","Data":"4adcf10126628f0ff0413a8f678e59ce5d8c66cb7234716b7c41855e90d65521"}
Jul 24 14:24:58.972110 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.971743    2527 scope.go:117] "RemoveContainer" containerID="491f261b05aea4381ba2e78cea967dcc025e6dc45e12fd0b93843a53529233fe"
Jul 24 14:24:58.973124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.973094    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5" event={"ID":"59822856-46d8-4f9c-be49-d625f0318763","Type":"ContainerDied","Data":"9847c3b1d64456aa4964c0161b107fe03ea7908bb2611e4ff445042b0030c98e"}
Jul 24 14:24:58.973124 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.973110    2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"
Jul 24 14:24:58.980997 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.980973    2527 scope.go:117] "RemoveContainer" containerID="85a0f3765dcced03f264616276497c3558c2442f3ad83db5e0a53254daba788e"
Jul 24 14:24:58.996332 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.996301    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:24:58.999932 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:58.999904    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-525009-7dbf44f6fd-fhr5m"]
Jul 24 14:24:59.015010 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:59.014976    2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:24:59.028575 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:24:59.023668    2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-525009-64d79dffdb-h5bn5"]
Jul 24 14:25:00.514335 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:00.514297    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59822856-46d8-4f9c-be49-d625f0318763" path="/var/lib/kubelet/pods/59822856-46d8-4f9c-be49-d625f0318763/volumes"
Jul 24 14:25:00.514735 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:00.514720    2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89639ce-58c1-4afa-881f-4d44900403e0" path="/var/lib/kubelet/pods/c89639ce-58c1-4afa-881f-4d44900403e0/volumes"
Jul 24 14:25:15.547920 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:15.547884    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-766d9c797c-wj8fp_975aabcd-e248-448b-a55c-82ef22695b11/manager/0.log"
Jul 24 14:25:15.895144 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:15.895116    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-52tsb_699813cd-8368-4600-93e8-dc3a0dfef5f4/copy-manifests/0.log"
Jul 24 14:25:15.901155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:15.901130    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-52tsb_699813cd-8368-4600-93e8-dc3a0dfef5f4/manager/0.log"
Jul 24 14:25:16.522964 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:16.522929    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b47444556-zp75g_a7a243d1-0d76-4f49-8591-387de9f80293/maas-api/0.log"
Jul 24 14:25:16.986537 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:16.986506    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-r7gbc_5df22bd4-67a2-4f30-b59d-742c70566fa3/postgres/0.log"
Jul 24 14:25:18.491748 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:18.491719    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7knf2_b01cd528-b42d-4d11-a626-838163db18e0/manager/0.log"
Jul 24 14:25:18.609155 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:18.609130    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8jjmp_94e55bc5-2306-49b9-bd02-b1e3d1504143/manager/0.log"
Jul 24 14:25:18.852093 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:18.852059    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k9q2k_7c0db250-bc4a-4986-9c7d-bfeee406ce43/registry-server/0.log"
Jul 24 14:25:20.234153 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:20.234124    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r26kw_021dd046-1cdc-4904-9e26-a0c95efb9ace/payload-pre-processing/0.log"
Jul 24 14:25:20.359425 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:20.359389    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:25:20.475282 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:20.475253    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d4b8fd45f-z8bhm_81c14746-f5a8-4bb3-8daa-93a2b84a3e6f/router/0.log"
Jul 24 14:25:20.826083 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:20.826045    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-w67pb_413017cd-cafd-459f-b573-9dff103a305b/main/0.log"
Jul 24 14:25:21.180240 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:21.180157    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcjm6z_3e8a9808-abb2-492c-90ea-075928b05eba/main/0.log"
Jul 24 14:25:21.421708 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:21.421673    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tbw54_f5efa39f-a6a5-4fb3-912d-28b35ffadff6/main/0.log"
Jul 24 14:25:29.637380 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:29.637344    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qq8rj_ac3c72de-350d-4005-9d29-bf40ae47ac5c/global-pull-secret-syncer/0.log"
Jul 24 14:25:29.729636 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:29.729609    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qnqt6_961928b2-f95d-4775-88c0-c80ab72bd7be/konnectivity-agent/0.log"
Jul 24 14:25:29.786180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:29.786148    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-199.ec2.internal_2f9f9e23f5a0fe49a38ad742dd8ed6c0/haproxy/0.log"
Jul 24 14:25:34.390955 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:34.390926    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7knf2_b01cd528-b42d-4d11-a626-838163db18e0/manager/0.log"
Jul 24 14:25:34.422057 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:34.422029    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8jjmp_94e55bc5-2306-49b9-bd02-b1e3d1504143/manager/0.log"
Jul 24 14:25:34.494968 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:34.494934    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k9q2k_7c0db250-bc4a-4986-9c7d-bfeee406ce43/registry-server/0.log"
Jul 24 14:25:36.574633 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:36.574601    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-k8tsd_8c235d3d-e3cf-47ba-bc21-baf3f442e5b8/cluster-monitoring-operator/0.log"
Jul 24 14:25:36.776307 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:36.776276    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97whs_862816b5-7915-40a1-b883-e1b84b70aae5/node-exporter/0.log"
Jul 24 14:25:36.802701 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:36.802656    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97whs_862816b5-7915-40a1-b883-e1b84b70aae5/kube-rbac-proxy/0.log"
Jul 24 14:25:36.839050 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:36.838983    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97whs_862816b5-7915-40a1-b883-e1b84b70aae5/init-textfile/0.log"
Jul 24 14:25:39.302395 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.302359    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-74xr7_964e98b6-6bfe-49cb-8e46-e64cc75ad9ae/networking-console-plugin/0.log"
Jul 24 14:25:39.960403 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.960369    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/2.log"
Jul 24 14:25:39.967486 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.967463    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-xqgfh_6d308633-b904-498a-b0b1-f8ca0a66fd0f/console-operator/1.log"
Jul 24 14:25:39.968099 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968081    2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"]
Jul 24 14:25:39.968470 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968458    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89639ce-58c1-4afa-881f-4d44900403e0" containerName="maas-api"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968472    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89639ce-58c1-4afa-881f-4d44900403e0" containerName="maas-api"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968480    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffd4034-f014-4133-a727-f4f361eba929" containerName="revoke-keys"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968485    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffd4034-f014-4133-a727-f4f361eba929" containerName="revoke-keys"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968502    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84989689-958d-4f13-8bb6-eff963fc2b72" containerName="maas-api"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968507    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="84989689-958d-4f13-8bb6-eff963fc2b72" containerName="maas-api"
Jul 24 14:25:39.968515 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968513    2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59822856-46d8-4f9c-be49-d625f0318763" containerName="payload-processing"
Jul 24 14:25:39.968699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968519    2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="59822856-46d8-4f9c-be49-d625f0318763" containerName="payload-processing"
Jul 24 14:25:39.968699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968575    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="84989689-958d-4f13-8bb6-eff963fc2b72" containerName="maas-api"
Jul 24 14:25:39.968699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968587    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89639ce-58c1-4afa-881f-4d44900403e0" containerName="maas-api"
Jul 24 14:25:39.968699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968593    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="59822856-46d8-4f9c-be49-d625f0318763" containerName="payload-processing"
Jul 24 14:25:39.968699 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.968599    2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffd4034-f014-4133-a727-f4f361eba929" containerName="revoke-keys"
Jul 24 14:25:39.971689 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.971672    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:39.973756 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.973736    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hh2p5\"/\"openshift-service-ca.crt\""
Jul 24 14:25:39.973890 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.973762    2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hh2p5\"/\"kube-root-ca.crt\""
Jul 24 14:25:39.974417 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.974399    2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hh2p5\"/\"default-dockercfg-gvc2t\""
Jul 24 14:25:39.980634 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:39.980608    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"]
Jul 24 14:25:40.073549 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.073507    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj76c\" (UniqueName: \"kubernetes.io/projected/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-kube-api-access-vj76c\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.073549 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.073552    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-lib-modules\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.073800 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.073590    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-podres\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.073800 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.073631    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-proc\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.073800 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.073662    2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-sys\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175105 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175067    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj76c\" (UniqueName: \"kubernetes.io/projected/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-kube-api-access-vj76c\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175281 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175125    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-lib-modules\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175281 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175170    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-podres\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175281 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175203    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-proc\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175281 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175260    2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-sys\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175555 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175323    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-lib-modules\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175555 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175330    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-podres\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175555 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175362    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-sys\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.175555 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.175339    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-proc\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.183564 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.183543    2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj76c\" (UniqueName: \"kubernetes.io/projected/4c31b496-0a8d-4fdc-b317-bfb7be617a4f-kube-api-access-vj76c\") pod \"perf-node-gather-daemonset-b2j6t\" (UID: \"4c31b496-0a8d-4fdc-b317-bfb7be617a4f\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.282285 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.282197    2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:40.434569 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.434537    2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"]
Jul 24 14:25:40.437997 ip-10-0-129-199 kubenswrapper[2527]: W0724 14:25:40.437960    2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c31b496_0a8d_4fdc_b317_bfb7be617a4f.slice/crio-c4d5059dece9bc76bf0353cf2f7f3bd206670a6b33841bfab7ae669ed3e05d38 WatchSource:0}: Error finding container c4d5059dece9bc76bf0353cf2f7f3bd206670a6b33841bfab7ae669ed3e05d38: Status 404 returned error can't find the container with id c4d5059dece9bc76bf0353cf2f7f3bd206670a6b33841bfab7ae669ed3e05d38
Jul 24 14:25:40.627063 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:40.627031    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-t6vhk_a2a0e067-907f-4aba-9145-5fc654769f71/download-server/0.log"
Jul 24 14:25:41.185954 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:41.185917    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t" event={"ID":"4c31b496-0a8d-4fdc-b317-bfb7be617a4f","Type":"ContainerStarted","Data":"18386ab9241f80fb23a280966d5dbe487c9f7b2f7bd3f997a0db530a6745d3d5"}
Jul 24 14:25:41.186180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:41.185978    2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t" event={"ID":"4c31b496-0a8d-4fdc-b317-bfb7be617a4f","Type":"ContainerStarted","Data":"c4d5059dece9bc76bf0353cf2f7f3bd206670a6b33841bfab7ae669ed3e05d38"}
Jul 24 14:25:41.186180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:41.186043    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:41.203399 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:41.203349    2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t" podStartSLOduration=2.20333447 podStartE2EDuration="2.20333447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:41.202856426 +0000 UTC m=+3105.244904313" watchObservedRunningTime="2026-07-24 14:25:41.20333447 +0000 UTC m=+3105.245382354"
Jul 24 14:25:41.207015 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:41.206987    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-r9pgq_4b8d0646-bb00-4c87-a2c8-4f5ece6de933/volume-data-source-validator/0.log"
Jul 24 14:25:42.261357 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:42.261329    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c946z_1a6d4f0a-0a6d-479f-a397-2eb4b324a169/dns/0.log"
Jul 24 14:25:42.283643 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:42.283614    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c946z_1a6d4f0a-0a6d-479f-a397-2eb4b324a169/kube-rbac-proxy/0.log"
Jul 24 14:25:42.480724 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:42.480687    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z26h4_a1e62ee6-1992-460f-ab28-59e9ceb5d89c/dns-node-resolver/0.log"
Jul 24 14:25:43.101249 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:43.101216    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9nhlx_8705eba6-6f23-4ad9-8a91-5cdee758f61a/node-ca/0.log"
Jul 24 14:25:44.472481 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:44.472422    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r26kw_021dd046-1cdc-4904-9e26-a0c95efb9ace/payload-pre-processing/0.log"
Jul 24 14:25:44.503552 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:44.503522    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:25:44.526632 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:44.526602    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d4b8fd45f-z8bhm_81c14746-f5a8-4bb3-8daa-93a2b84a3e6f/router/0.log"
Jul 24 14:25:45.397743 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:45.397714    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hggbw_49397988-6b3a-4429-a7c0-b700fd2b1088/serve-healthcheck-canary/0.log"
Jul 24 14:25:45.933968 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:45.933933    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7896g_1587d831-aefd-4241-8f63-e0a31867f65d/insights-operator/0.log"
Jul 24 14:25:45.934787 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:45.934770    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7896g_1587d831-aefd-4241-8f63-e0a31867f65d/insights-operator/1.log"
Jul 24 14:25:46.223913 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:46.223837    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvln4_de81ac99-198b-421c-88d4-998d005dcb61/kube-rbac-proxy/0.log"
Jul 24 14:25:46.254136 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:46.254111    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvln4_de81ac99-198b-421c-88d4-998d005dcb61/exporter/0.log"
Jul 24 14:25:46.286901 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:46.286872    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvln4_de81ac99-198b-421c-88d4-998d005dcb61/extractor/0.log"
Jul 24 14:25:47.202554 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:47.202526    2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-b2j6t"
Jul 24 14:25:49.437404 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:49.437376    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-766d9c797c-wj8fp_975aabcd-e248-448b-a55c-82ef22695b11/manager/0.log"
Jul 24 14:25:49.577393 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:49.577365    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-52tsb_699813cd-8368-4600-93e8-dc3a0dfef5f4/manager/0.log"
Jul 24 14:25:49.597349 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:49.597322    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-52tsb_699813cd-8368-4600-93e8-dc3a0dfef5f4/copy-manifests/0.log"
Jul 24 14:25:50.720180 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:50.720149    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r26kw_021dd046-1cdc-4904-9e26-a0c95efb9ace/payload-pre-processing/0.log"
Jul 24 14:25:50.739196 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:50.739166    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:25:50.752308 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:50.752281    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d4b8fd45f-z8bhm_81c14746-f5a8-4bb3-8daa-93a2b84a3e6f/router/0.log"
Jul 24 14:25:51.023941 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:51.023909    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d64c44479-rxtc2_9b73012c-00af-45d3-8b00-e17fbaa75141/manager/0.log"
Jul 24 14:25:52.415022 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:52.414992    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r26kw_021dd046-1cdc-4904-9e26-a0c95efb9ace/payload-pre-processing/0.log"
Jul 24 14:25:52.432668 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:52.432637    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ghdbf_70aae07d-b447-4198-9132-48c66e275e68/payload-processing/0.log"
Jul 24 14:25:52.446326 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:52.446292    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d4b8fd45f-z8bhm_81c14746-f5a8-4bb3-8daa-93a2b84a3e6f/router/0.log"
Jul 24 14:25:56.116600 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:56.116563    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-5cp48_4407ce96-9260-42e3-aead-f8006c87f3b1/migrator/0.log"
Jul 24 14:25:56.146352 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:56.146327    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-5cp48_4407ce96-9260-42e3-aead-f8006c87f3b1/graceful-termination/0.log"
Jul 24 14:25:56.508497 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:56.508454    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf_65b2a17c-563e-4e60-868b-41f51530c308/kube-storage-version-migrator-operator/1.log"
Jul 24 14:25:56.509503 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:56.509480    2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-65sqf_65b2a17c-563e-4e60-868b-41f51530c308/kube-storage-version-migrator-operator/0.log"
Jul 24 14:25:56.886167 ip-10-0-129-199 kubenswrapper[2527]: I0724 14:25:56.886135    2527 scope.go:117] "RemoveContainer" containerID="2813d63a0d61286dc67c195c851e4bfcb55075990f4fde2c112d216d6235858c"