Jul 31 18:43:41.526113 ip-10-0-128-163 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 18:43:41.968183 ip-10-0-128-163 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 18:43:41.968183 ip-10-0-128-163 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 18:43:41.968183 ip-10-0-128-163 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 18:43:41.968183 ip-10-0-128-163 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 18:43:41.968183 ip-10-0-128-163 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 18:43:41.969134 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.969046    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 18:43:41.973326 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973305    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.973326 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973322    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.973326 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973327    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.973326 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973331    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973336    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973340    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973344    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973347    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973351    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973356    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973360    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973367    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973372    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973375    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973383    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973387    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973392    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973396    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973400    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973404    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973407    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973412    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973416    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.973610 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973420    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973424    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973427    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973431    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973435    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973456    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973461    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973465    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973469    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973473    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973477    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973481    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973485    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973490    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973494    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973498    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973503    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973507    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973511    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973516    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.974420 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973520    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973524    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973529    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973533    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973545    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973550    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973554    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973558    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973562    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973566    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973570    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973574    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973579    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973583    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973587    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973592    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973597    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973601    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973606    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.975269 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973610    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973614    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973618    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973623    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973627    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973631    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973635    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973640    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973644    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973649    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973653    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973658    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973662    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973668    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973672    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973677    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973681    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973686    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973690    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973694    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.976110 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973698    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973702    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973706    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973710    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973716    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973720    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973724    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973728    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973733    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973737    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973744    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973750    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973755    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973759    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973764    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973769    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.973775    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974937    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974948    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.976976 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974954    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974959    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974964    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974968    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974973    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974977    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974981    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974986    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974991    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974995    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.974999    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975003    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975010    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975016    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975021    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975025    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975030    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975034    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975039    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.977870 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975044    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975051    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975056    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975061    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975066    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975070    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975074    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975079    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975083    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975088    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975092    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975096    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975102    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975107    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975111    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975115    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975119    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975123    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975128    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975132    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.978542 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975136    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975141    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975145    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975149    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975153    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975159    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975163    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975168    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975172    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975177    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975181    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975185    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975189    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975193    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975197    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975201    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975205    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975210    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975214    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975218    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.979131 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975222    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975228    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975232    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975236    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975240    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975245    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975249    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975253    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975257    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975262    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975266    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975270    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975274    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975277    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975286    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975291    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975295    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975299    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975304    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975308    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.979663 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975312    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975316    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975320    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975324    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975328    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975332    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975336    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975340    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975344    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975348    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975352    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975357    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975361    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975365    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975369    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975375    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975381    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.975386    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976063    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.980357 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976072    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976076    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976081    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976085    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976089    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976094    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976099    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976104    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976108    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976112    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976116    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976120    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976125    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976130    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976135    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976139    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976143    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976148    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976152    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.980973 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976156    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976161    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976165    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976169    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976173    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976177    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976182    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976186    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976191    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976195    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976199    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976204    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976208    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976213    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976217    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976222    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976226    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976230    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976234    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976238    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.981465 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976242    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976247    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976251    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976256    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976260    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976264    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976268    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976272    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976276    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976280    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976284    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976288    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976292    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976296    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976300    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976304    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976309    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976313    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976321    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.982022 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976327    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976331    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976336    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976340    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976344    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976348    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976352    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976359    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976364    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976369    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976374    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976378    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976382    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976387    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976390    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976400    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976408    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976412    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976416    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.982552 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976420    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976425    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976429    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976433    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976459    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976465    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976470    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976475    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976479    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976483    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976487    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976491    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976496    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976500    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976504    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976508    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976513    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976517    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976529    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976533    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.976537    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.983020 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976640    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976671    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976684    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976693    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976700    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976705    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976712    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976719    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976734    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976739    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976745    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976751    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976756    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976760    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976765    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976769    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976773    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976778    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976789    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976794    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976799    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976804    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976809    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976815    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 18:43:41.983562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976820    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976825    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976829    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976834    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976838    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976843    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976848    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976852    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976858    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976863    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976868    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976873    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976877    2557 flags.go:64] FLAG: --enable-server="true"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976882    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976891    2557 flags.go:64] FLAG: --event-burst="100"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976896    2557 flags.go:64] FLAG: --event-qps="50"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976902    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976917    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976923    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976929    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976934    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976939    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976944    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976949    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976954    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 18:43:41.984139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976959    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976963    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976968    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976972    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976976    2557 flags.go:64] FLAG: --feature-gates=""
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976982    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976987    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976991    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.976996    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977001    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977005    2557 flags.go:64] FLAG: --help="false"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977010    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-128-163.ec2.internal"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977015    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977020    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977024    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977030    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977035    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977040    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977045    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977049    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977053    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977058    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977063    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977070    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 31 18:43:41.984779 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977074    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977080    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977084    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977089    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977093    2557 flags.go:64] FLAG: --lock-file=""
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977098    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977102    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977109    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977117    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977122    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977126    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977130    2557 flags.go:64] FLAG: --logging-format="text"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977135    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977140    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977145    2557 flags.go:64] FLAG: --manifest-url=""
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977149    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977156    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977160    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977167    2557 flags.go:64] FLAG: --max-pods="110"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977171    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977176    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977180    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977185    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977190    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977194    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 18:43:41.985420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977199    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977209    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977214    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977219    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977224    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977228    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977238    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977243    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977248    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977253    2557 flags.go:64] FLAG: --port="10250"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977258    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977262    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff1c128d65d132b5"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977268    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977272    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977277    2557 flags.go:64] FLAG: --register-node="true"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977283    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977290    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977294    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977299    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977303    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977309    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977314    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977319    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977323    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977327    2557 flags.go:64] FLAG: --runonce="false"
Jul 31 18:43:41.986068 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977332    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977337    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977341    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977346    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977351    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977356    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977361    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977365    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977370    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977375    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977379    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977383    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977388    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977396    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977427    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977456    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977462    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977469    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977476    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977481    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977485    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977490    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977496    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977501    2557 flags.go:64] FLAG: --v="2"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977507    2557 flags.go:64] FLAG: --version="false"
Jul 31 18:43:41.986729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977514    2557 flags.go:64] FLAG: --vmodule=""
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977520    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.977525    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977739    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977746    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977751    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977756    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977761    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977766    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977772    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977779    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977785    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977790    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977794    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977799    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977803    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977808    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977812    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977816    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.987348 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977820    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977826    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977831    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977835    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977840    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977844    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977849    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977853    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977857    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977861    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977869    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977873    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977877    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977880    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977885    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977888    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977893    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977897    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977901    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977905    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.987833 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977909    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977913    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977917    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977922    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977926    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977930    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977934    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977938    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977942    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977946    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977950    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977954    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977958    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977962    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977967    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977971    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977976    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977980    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977984    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977989    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.988323 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977993    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.977997    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978003    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978007    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978011    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978015    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978020    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978026    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978031    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978035    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978039    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978043    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978047    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978051    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978056    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978060    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978064    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978068    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978072    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.988872 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978077    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978081    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978085    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978089    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978094    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978098    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978102    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978106    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978110    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978114    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978118    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978123    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978127    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978133    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978137    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978142    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978147    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978151    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978155    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978159    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.989351 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978163    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978169    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978174    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.978179    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.978960    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.987292    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.987310    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.987327    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.987333    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987410    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987416    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987419    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987422    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987425    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.989939 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987428    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987431    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987434    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987450    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987453    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987457    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987459    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987462    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987465    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987474    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987479    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987490    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987493    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987496    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987499    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987502    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987504    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987507    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987509    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.990292 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987512    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987515    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987517    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987520    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987523    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987526    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987529    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987531    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987534    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987536    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987539    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987541    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987544    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987546    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987549    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987551    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987557    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987560    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987562    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.990775 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987565    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987567    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987569    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987573    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987577    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987580    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987588    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987591    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987593    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987595    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987598    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987601    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987603    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987605    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987608    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987610    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987613    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987615    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987619    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987621    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.991240 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987624    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987626    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987629    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987631    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987634    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987636    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987638    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987641    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987643    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987645    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987648    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987650    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987652    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987655    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987657    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987659    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987662    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987664    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987666    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987674    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.991786 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987677    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987679    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987682    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987684    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987687    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987690    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987692    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987697    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987700    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987703    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987706    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987709    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987711    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987714    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987716    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.992312 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987718    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.987723    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987883    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987888    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987891    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987894    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987897    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987900    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987902    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987905    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987908    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987910    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987913    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987915    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987917    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 18:43:41.992710 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987920    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987922    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987930    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987932    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987936    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987940    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987944    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987946    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987949    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987951    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987954    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987956    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987959    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987962    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987964    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987967    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987969    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987971    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987974    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987976    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 18:43:41.993071 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987978    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987981    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987984    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987986    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987988    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987991    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987993    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987995    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.987998    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988000    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988003    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988005    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988007    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988010    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988012    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988020    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988022    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988026    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988029    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988031    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 18:43:41.993598 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988034    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988036    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988039    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988041    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988044    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988046    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988049    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988051    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988054    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988056    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988058    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988061    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988063    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988066    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988069    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988072    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988076    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988079    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988082    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 18:43:41.994084 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988085    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988087    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988091    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988095    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988097    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988100    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988103    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988105    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988108    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988117    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988121    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988124    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988126    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988129    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988132    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988135    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988137    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988140    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988142    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 18:43:41.994569 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988144    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988147    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988149    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988152    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988154    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988156    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988159    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:41.988161    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.988166    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.988302    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.990950    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.992698    2557 server.go:1019] "Starting client certificate rotation"
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.992803    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 18:43:41.995025 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.993790    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 18:43:41.995904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.995881    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 18:43:41.997264 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:41.997242    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 18:43:42.010683 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.010663    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 31 18:43:42.014647 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.014631    2557 log.go:25] "Validated CRI v1 image API"
Jul 31 18:43:42.015487 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.015472    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 18:43:42.018112 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.018086    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b0a61d3a-52fa-4662-858f-4f7c96b3aebb:/dev/nvme0n1p3 eb49ca76-16e4-4107-af4d-6c6cf6141c22:/dev/nvme0n1p4]
Jul 31 18:43:42.018172 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.018111    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 18:43:42.024167 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.024056    2557 manager.go:216] Machine: {Timestamp:2026-07-31 18:43:42.021772816 +0000 UTC m=+0.369354687 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100170 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb1a76491e37b625444bccf2e9260 SystemUUID:ec2eb1a7-6491-e37b-6254-44bccf2e9260 BootID:9803e528-0e75-49de-8f5b-b5091c52d29b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:68:15:f3:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:68:15:f3:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:74:00:1a:00:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 18:43:42.024167 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.024161    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 18:43:42.024268 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.024242    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 18:43:42.028107 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.028077    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 18:43:42.028275 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.028106    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 18:43:42.028416 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.028401    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 18:43:42.028996 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.028984    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 18:43:42.029040 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.029000    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 18:43:42.029040 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.029008    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 18:43:42.029040 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.029019    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 18:43:42.030891 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.030881    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 18:43:42.036320 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036309    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 18:43:42.036436 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036427    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 18:43:42.036483 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036470    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 18:43:42.036483 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036481    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 18:43:42.036551 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036495    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 18:43:42.036551 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036504    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 31 18:43:42.036551 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036513    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 18:43:42.036866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036854    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 18:43:42.036900 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.036873    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 18:43:42.039181 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.039163    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 18:43:42.041014 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041002    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 18:43:42.041057 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041032    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 18:43:42.041812 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041799    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041815    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041821    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041827    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041834    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041839    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041866    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041872    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041879    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 18:43:42.041883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041884    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 18:43:42.042106 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041902    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 18:43:42.042106 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.041909    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 18:43:42.043749 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.043737    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 18:43:42.043796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.043752    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 18:43:42.046124 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.046106    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 18:43:42.046202 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.046153    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 18:43:42.046202 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.046158    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 18:43:42.047181 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.047165    2557 server.go:1268] "Started kubelet"
Jul 31 18:43:42.047265 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.047241    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 18:43:42.051370 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.047248    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 18:43:42.051471 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.051403    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 18:43:42.052034 ip-10-0-128-163 systemd[1]: Started Kubernetes Kubelet.
Jul 31 18:43:42.052846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.052829    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 18:43:42.054038 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.054022    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 18:43:42.057897 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.057876    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 18:43:42.058181 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.057071    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-163.ec2.internal.18c774179d427207  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-163.ec2.internal,UID:ip-10-0-128-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-163.ec2.internal,},FirstTimestamp:2026-07-31 18:43:42.047138311 +0000 UTC m=+0.394720182,LastTimestamp:2026-07-31 18:43:42.047138311 +0000 UTC m=+0.394720182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-163.ec2.internal,}"
Jul 31 18:43:42.061771 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.061721    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 18:43:42.061771 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.061725    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 18:43:42.061891 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.061789    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.062555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062537    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 18:43:42.062555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062539    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 18:43:42.062682 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062565    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 18:43:42.062682 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062645    2557 factory.go:55] Registering systemd factory
Jul 31 18:43:42.062682 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062659    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 31 18:43:42.062849 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062833    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 18:43:42.062849 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062850    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 18:43:42.062955 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062895    2557 factory.go:153] Registering CRI-O factory
Jul 31 18:43:42.062955 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062917    2557 factory.go:223] Registration of the crio container factory successfully
Jul 31 18:43:42.063063 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062966    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 18:43:42.063063 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.062988    2557 factory.go:103] Registering Raw factory
Jul 31 18:43:42.063063 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.063000    2557 manager.go:1196] Started watching for new ooms in manager
Jul 31 18:43:42.064072 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.064059    2557 manager.go:349] Starting recovery of all containers
Jul 31 18:43:42.072247 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.072226    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 18:43:42.072391 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.072337    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 18:43:42.074473 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.074452    2557 manager.go:354] Recovery completed
Jul 31 18:43:42.078477 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.078462    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.080365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.080323    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.080468 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.080382    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.080468 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.080398    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.081065 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081051    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 18:43:42.081065 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081064    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 18:43:42.081139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081079    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 18:43:42.081139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081058    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.081734 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081720    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mhnkz"
Jul 31 18:43:42.081839 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081825    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.081905 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081855    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.081905 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.081870    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.082143 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.082130    2557 policy_none.go:49] "None policy: Start"
Jul 31 18:43:42.082201 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.082149    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 18:43:42.082201 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.082163    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 18:43:42.083075 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.083062    2557 policy_none.go:47] "Start"
Jul 31 18:43:42.089475 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.089308    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mhnkz"
Jul 31 18:43:42.092334 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.092261    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-163.ec2.internal.18c774179f3d7329  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-163.ec2.internal,UID:ip-10-0-128-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-163.ec2.internal,},FirstTimestamp:2026-07-31 18:43:42.080365353 +0000 UTC m=+0.427947229,LastTimestamp:2026-07-31 18:43:42.080365353 +0000 UTC m=+0.427947229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-163.ec2.internal,}"
Jul 31 18:43:42.120996 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.120979    2557 manager.go:337] "Starting Device Plugin manager"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.121012    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121022    2557 server.go:87] "Starting device plugin registration server"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121362    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121375    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121495    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121606    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.121616    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.122208    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 18:43:42.132713 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.122260    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.175828 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.175799    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 18:43:42.177155 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.177140    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 18:43:42.177227 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.177175    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 18:43:42.177227 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.177196    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 18:43:42.177227 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.177224    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 18:43:42.179951 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.179930    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 18:43:42.222377 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.222295    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.223305 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.223286    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.223402 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.223323    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.223402 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.223340    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.223402 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.223374    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.230483 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.230465    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.230566 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.230492    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-163.ec2.internal\": node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.260006 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.259979    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.278152 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.278122    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"]
Jul 31 18:43:42.278224 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.278192    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.278990 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.278968    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.279089 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.279001    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.279089 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.279016    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.280317 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.280301    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.280471 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.280455    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.280528 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.280491    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.281033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.281019    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.281090 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.281045    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.281090 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.281055    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.282093 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282080    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.282147 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282105    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:42.282196 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282183    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.282255 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282205    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.282255 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282219    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.282757 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282740    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:42.282824 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282768    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:42.282824 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.282780    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:42.309234 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.309207    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-163.ec2.internal\" not found" node="ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.312667 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.312651    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-163.ec2.internal\" not found" node="ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.360106 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.360080    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.364452 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.364423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.364529 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.364462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.364529 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.364480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda49057b793c5c62334b176ef9bc360-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"fda49057b793c5c62334b176ef9bc360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.460196 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.460152    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.465561 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.465622 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.465622 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda49057b793c5c62334b176ef9bc360-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"fda49057b793c5c62334b176ef9bc360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.465714 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.465714 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465634    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fda49057b793c5c62334b176ef9bc360-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"fda49057b793c5c62334b176ef9bc360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.465714 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.465677    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6059c7ee63b4af2d85012a3466eef7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"9d6059c7ee63b4af2d85012a3466eef7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.561068 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.560994    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.611500 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.611469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.616227 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.615517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:42.661546 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.661511    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.761962 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.761913    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.862413 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.862332    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.962893 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:42.962867    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:42.992149 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.992118    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 18:43:42.992610 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:42.992267    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 18:43:43.063007 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:43.062981    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:43.094157 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.094108    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 18:38:42 +0000 UTC" deadline="2028-04-28 05:11:36.771542088 +0000 UTC"
Jul 31 18:43:43.094157 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.094147    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15274h27m53.677398183s"
Jul 31 18:43:43.159316 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.159249    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 18:43:43.163496 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:43.163478    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:43.259908 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.259882    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 18:43:43.261731 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.261714    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 18:43:43.261805 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.261754    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 18:43:43.261805 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.261769    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
Jul 31 18:43:43.264031 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:43.264010    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
Jul 31 18:43:43.308333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.308312    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 18:43:43.311645 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:43.311620    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda49057b793c5c62334b176ef9bc360.slice/crio-941b4aa7cf70940db291576ee11ce4f26b3a6610610b81c23f2ad31ff117ae95 WatchSource:0}: Error finding container 941b4aa7cf70940db291576ee11ce4f26b3a6610610b81c23f2ad31ff117ae95: Status 404 returned error can't find the container with id 941b4aa7cf70940db291576ee11ce4f26b3a6610610b81c23f2ad31ff117ae95
Jul 31 18:43:43.312692 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:43.312671    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6059c7ee63b4af2d85012a3466eef7.slice/crio-805f0b192bb3bec1a3adcfb4db53977eadfad18175e18c5fe1c402bdcd765553 WatchSource:0}: Error finding container 805f0b192bb3bec1a3adcfb4db53977eadfad18175e18c5fe1c402bdcd765553: Status 404 returned error can't find the container with id 805f0b192bb3bec1a3adcfb4db53977eadfad18175e18c5fe1c402bdcd765553
Jul 31 18:43:43.316208 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.316186    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 18:43:43.363264 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.363241    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:43.374669 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.374645    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 18:43:43.375600 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.375586    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
Jul 31 18:43:43.393259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.393232    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 18:43:43.588460 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.588350    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 18:43:43.851946 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:43.851860    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 18:43:44.038456 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.038421    2557 apiserver.go:52] "Watching apiserver"
Jul 31 18:43:44.050651 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.050623    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 18:43:44.051772 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.051734    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-68swk","openshift-network-operator/iptables-alerter-hgsr2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx","openshift-dns/node-resolver-rprxx","openshift-image-registry/node-ca-d66d8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal","openshift-multus/network-metrics-daemon-stg44","openshift-network-diagnostics/network-check-target-5gnwc","openshift-ovn-kubernetes/ovnkube-node-zkck9","kube-system/global-pull-secret-syncer-fhf8w","kube-system/konnectivity-agent-99v4g","kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7tgm2","openshift-multus/multus-4vcn4"]
Jul 31 18:43:44.054717 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.054693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.054846 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.054783    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:44.056767 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.056742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.059139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.059097    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.059589 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.059538    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wqnfg\""
Jul 31 18:43:44.059780 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.059764    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.060057 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.060036    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.061599 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.061573    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.062136 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.062114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 18:43:44.062231 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.062134    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.062314 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.062298    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bt6x\""
Jul 31 18:43:44.062469 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.062453    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 18:43:44.062944 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.062927    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.064209 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.063693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.064332 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.064311    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.064450 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.064431    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.064632 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.064615    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qn59\""
Jul 31 18:43:44.066561 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.066543    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.066968 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.066948    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 18:43:44.067177 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.067156    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.067392 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.067373    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wkq7c\""
Jul 31 18:43:44.068974 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.068952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.073431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8de46e-3b7b-4896-a7c4-2256edda4630-host\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.073549 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5958m\" (UniqueName: \"kubernetes.io/projected/9f8de46e-3b7b-4896-a7c4-2256edda4630-kube-api-access-5958m\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.073549 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073477    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.073549 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-systemd\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073551    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgbk\" (UniqueName: \"kubernetes.io/projected/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-kube-api-access-tmgbk\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-conf\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073602    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-registration-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073628    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-device-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4cm\" (UniqueName: \"kubernetes.io/projected/467a45a8-1a97-4162-8310-40ab1a20a1a5-kube-api-access-rs4cm\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073704    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.073727 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v9n\" (UniqueName: \"kubernetes.io/projected/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-kube-api-access-d6v9n\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysconfig\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073792    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ed081d-85b1-41da-af82-2caff6b85486-hosts-file\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073821    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjsc\" (UniqueName: \"kubernetes.io/projected/f7ed081d-85b1-41da-af82-2caff6b85486-kube-api-access-xtjsc\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073844    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-sys\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-var-lib-kubelet\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073896    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-modprobe-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-tuned\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-tmp\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-sys-fs\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.073993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed081d-85b1-41da-af82-2caff6b85486-tmp-dir\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-kubernetes\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074044    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-run\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074129 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-host\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-socket-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.074924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8de46e-3b7b-4896-a7c4-2256edda4630-serviceca\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.074924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-lib-modules\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.074924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074392    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.074924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.074744    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 18:43:44.075202 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.075178    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mldd6\""
Jul 31 18:43:44.075428 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.075411    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 18:43:44.075669 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.075652    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.077539 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.076504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.077539 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.077305    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 18:43:44.077791 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.077632    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 18:43:44.078275 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.077859    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.078275 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.078089    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.079512 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.079492    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 18:43:44.079885 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.079866    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 18:43:44.080128 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.080083    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-thmd4\""
Jul 31 18:43:44.080460 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.080442    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 18:43:44.080725 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.080663    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 18:43:44.080849 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.080671    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-27jb9\""
Jul 31 18:43:44.080989 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.080969    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.081055 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.081035    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:44.083018 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.082997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.086656    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.086752    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.086898    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.086913    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.087156    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 18:43:44.087927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.087172    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mnl8l\""
Jul 31 18:43:44.092613 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.092588    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 18:43:44.092800 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.092780    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2jtr\""
Jul 31 18:43:44.092996 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.092978    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 18:43:44.093069 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.092994    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 18:43:44.112681 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.112648    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jp5nb"
Jul 31 18:43:44.121851 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.121824    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jp5nb"
Jul 31 18:43:44.163837 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.163745    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 18:43:44.174672 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174645    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-conf\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4cm\" (UniqueName: \"kubernetes.io/projected/467a45a8-1a97-4162-8310-40ab1a20a1a5-kube-api-access-rs4cm\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/4505fee6-5b31-49a9-8125-2c57e9ae7be4-kube-api-access-ztrtt\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-bin\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174788    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-os-release\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.174817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-system-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174865    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-conf\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ed081d-85b1-41da-af82-2caff6b85486-hosts-file\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb4598ba-52ca-438c-bfff-f680e640210d-iptables-alerter-script\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.174991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ed081d-85b1-41da-af82-2caff6b85486-hosts-file\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.175033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-tuned\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-tmp\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-sys-fs\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-ovn\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-system-cni-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-bin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed081d-85b1-41da-af82-2caff6b85486-tmp-dir\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175259    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175292 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-run\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-kubelet\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13caa362-adf4-45d4-a4da-fadf86931e04-ovn-node-metrics-cert\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-sys-fs\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175390    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-daemon-config\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb4598ba-52ca-438c-bfff-f680e640210d-host-slash\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8de46e-3b7b-4896-a7c4-2256edda4630-serviceca\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-lib-modules\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysctl-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-node-log\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175361    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 18:43:44.175601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175602    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-run\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-lib-modules\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cnibin\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-etc-kubernetes\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-systemd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-config\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cni-binary-copy\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgbk\" (UniqueName: \"kubernetes.io/projected/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-kube-api-access-tmgbk\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-env-overrides\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89e00fd6-040c-4723-b241-289e17f0ef8e-agent-certs\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.175982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-multus-certs\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-registration-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176042    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8de46e-3b7b-4896-a7c4-2256edda4630-serviceca\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-device-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176073    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.176282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-netns\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-netns\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v9n\" (UniqueName: \"kubernetes.io/projected/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-kube-api-access-d6v9n\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysconfig\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89e00fd6-040c-4723-b241-289e17f0ef8e-konnectivity-ca\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-kubelet-config\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjsc\" (UniqueName: \"kubernetes.io/projected/f7ed081d-85b1-41da-af82-2caff6b85486-kube-api-access-xtjsc\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-sys\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-var-lib-kubelet\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176416    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-log-socket\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-conf-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176494    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-modprobe-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-netd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cnibin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-socket-dir-parent\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.177162 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-kubernetes\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176638    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-host\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-socket-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-k8s-cni-cncf-io\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzw7c\" (UniqueName: \"kubernetes.io/projected/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-kube-api-access-dzw7c\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-sysconfig\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-device-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-registration-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.176926    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.176969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.177014    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:43:44.676978158 +0000 UTC m=+3.024560033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-sys\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177036    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-var-lib-kubelet\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-systemd-units\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-slash\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178099 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-etc-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-script-lib\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/13caa362-adf4-45d4-a4da-fadf86931e04-kube-api-access-zqphq\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-modprobe-d\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-os-release\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177239    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-host\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-kubernetes\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/467a45a8-1a97-4162-8310-40ab1a20a1a5-socket-dir\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgsd\" (UniqueName: \"kubernetes.io/projected/bb4598ba-52ca-438c-bfff-f680e640210d-kube-api-access-2vgsd\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-dbus\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8de46e-3b7b-4896-a7c4-2256edda4630-host\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5958m\" (UniqueName: \"kubernetes.io/projected/9f8de46e-3b7b-4896-a7c4-2256edda4630-kube-api-access-5958m\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-systemd\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-hostroot\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177638    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-systemd\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8de46e-3b7b-4896-a7c4-2256edda4630-host\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-var-lib-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.178981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-binary-copy\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.179902 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-multus\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.179902 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.177821    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-kubelet\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.179902 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.178104    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ed081d-85b1-41da-af82-2caff6b85486-tmp-dir\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.179902 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.179548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-etc-tuned\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.181058 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.181041    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-tmp\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.183713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.183651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"9d6059c7ee63b4af2d85012a3466eef7","Type":"ContainerStarted","Data":"805f0b192bb3bec1a3adcfb4db53977eadfad18175e18c5fe1c402bdcd765553"}
Jul 31 18:43:44.184866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.184843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" event={"ID":"fda49057b793c5c62334b176ef9bc360","Type":"ContainerStarted","Data":"941b4aa7cf70940db291576ee11ce4f26b3a6610610b81c23f2ad31ff117ae95"}
Jul 31 18:43:44.187753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.187729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4cm\" (UniqueName: \"kubernetes.io/projected/467a45a8-1a97-4162-8310-40ab1a20a1a5-kube-api-access-rs4cm\") pod \"aws-ebs-csi-driver-node-t2qjx\" (UID: \"467a45a8-1a97-4162-8310-40ab1a20a1a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.189316 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.189298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgbk\" (UniqueName: \"kubernetes.io/projected/6435d2b7-3eb8-4ee0-ab81-6c7862a7d376-kube-api-access-tmgbk\") pod \"tuned-7tgm2\" (UID: \"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376\") " pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.189825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.189805    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5958m\" (UniqueName: \"kubernetes.io/projected/9f8de46e-3b7b-4896-a7c4-2256edda4630-kube-api-access-5958m\") pod \"node-ca-d66d8\" (UID: \"9f8de46e-3b7b-4896-a7c4-2256edda4630\") " pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.189931 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.189917    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjsc\" (UniqueName: \"kubernetes.io/projected/f7ed081d-85b1-41da-af82-2caff6b85486-kube-api-access-xtjsc\") pod \"node-resolver-rprxx\" (UID: \"f7ed081d-85b1-41da-af82-2caff6b85486\") " pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.193651 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.193627    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v9n\" (UniqueName: \"kubernetes.io/projected/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-kube-api-access-d6v9n\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.278647 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-node-log\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cnibin\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-etc-kubernetes\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-node-log\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-systemd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-config\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278780    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cni-binary-copy\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.278815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cnibin\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-env-overrides\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89e00fd6-040c-4723-b241-289e17f0ef8e-agent-certs\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-multus-certs\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-netns\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-netns\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89e00fd6-040c-4723-b241-289e17f0ef8e-konnectivity-ca\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-kubelet-config\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-log-socket\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.279087 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-conf-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-netd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cnibin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-socket-dir-parent\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-k8s-cni-cncf-io\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw7c\" (UniqueName: \"kubernetes.io/projected/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-kube-api-access-dzw7c\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-systemd-units\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-slash\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-etc-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-script-lib\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/13caa362-adf4-45d4-a4da-fadf86931e04-kube-api-access-zqphq\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-os-release\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279429    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cni-binary-copy\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgsd\" (UniqueName: \"kubernetes.io/projected/bb4598ba-52ca-438c-bfff-f680e640210d-kube-api-access-2vgsd\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279494    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-conf-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.279555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-dbus\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-cnibin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-hostroot\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279582    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-socket-dir-parent\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-var-lib-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-k8s-cni-cncf-io\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279643    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-systemd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279647    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-binary-copy\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-multus\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-kubelet\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/4505fee6-5b31-49a9-8125-2c57e9ae7be4-kube-api-access-ztrtt\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-bin\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-os-release\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-system-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.280345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb4598ba-52ca-438c-bfff-f680e640210d-iptables-alerter-script\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-config\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.278781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-etc-kubernetes\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-os-release\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-systemd-units\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280146    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-slash\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280190    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-etc-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-dbus\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-ovn\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280383    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-system-cni-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-bin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-kubelet\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13caa362-adf4-45d4-a4da-fadf86931e04-ovn-node-metrics-cert\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280482    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-env-overrides\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-daemon-config\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb4598ba-52ca-438c-bfff-f680e640210d-host-slash\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280612    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13caa362-adf4-45d4-a4da-fadf86931e04-ovnkube-script-lib\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280853    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-daemon-config\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-run-ovn\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-system-cni-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280957    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-bin\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.280981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-kubelet\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281379    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df4356d7-969b-4b93-a3e6-3fb45690cd29-kubelet-config\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281423    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-multus-certs\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-netns\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281490    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-run-netns\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-netd\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-log-socket\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.281825 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-binary-copy\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-var-lib-openvswitch\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.279610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-hostroot\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-os-release\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281956    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-cni-multus\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.281999    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-host-var-lib-kubelet\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89e00fd6-040c-4723-b241-289e17f0ef8e-konnectivity-ca\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282095    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-multus-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-cni-bin\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.282234    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13caa362-adf4-45d4-a4da-fadf86931e04-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.282289    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:44.78227031 +0000 UTC m=+3.129852175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282371    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4505fee6-5b31-49a9-8125-2c57e9ae7be4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-system-cni-dir\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4505fee6-5b31-49a9-8125-2c57e9ae7be4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.282804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb4598ba-52ca-438c-bfff-f680e640210d-host-slash\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.283639 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.282904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb4598ba-52ca-438c-bfff-f680e640210d-iptables-alerter-script\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.285424 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.285394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13caa362-adf4-45d4-a4da-fadf86931e04-ovn-node-metrics-cert\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.286210 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.286184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89e00fd6-040c-4723-b241-289e17f0ef8e-agent-certs\") pod \"konnectivity-agent-99v4g\" (UID: \"89e00fd6-040c-4723-b241-289e17f0ef8e\") " pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.288256 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.288234    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:43:44.288356 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.288261    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:43:44.288356 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.288281    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:44.288475 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.288358    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:44.788321933 +0000 UTC m=+3.135903809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:44.292424 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.292353    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgsd\" (UniqueName: \"kubernetes.io/projected/bb4598ba-52ca-438c-bfff-f680e640210d-kube-api-access-2vgsd\") pod \"iptables-alerter-hgsr2\" (UID: \"bb4598ba-52ca-438c-bfff-f680e640210d\") " pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.293352 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.293331    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/13caa362-adf4-45d4-a4da-fadf86931e04-kube-api-access-zqphq\") pod \"ovnkube-node-zkck9\" (UID: \"13caa362-adf4-45d4-a4da-fadf86931e04\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.294413 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.294381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/4505fee6-5b31-49a9-8125-2c57e9ae7be4-kube-api-access-ztrtt\") pod \"multus-additional-cni-plugins-68swk\" (UID: \"4505fee6-5b31-49a9-8125-2c57e9ae7be4\") " pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.295042 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.295012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw7c\" (UniqueName: \"kubernetes.io/projected/06af2d8b-c23f-4b50-9cc4-5da5cc5b3880-kube-api-access-dzw7c\") pod \"multus-4vcn4\" (UID: \"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880\") " pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.374301 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.374221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7tgm2"
Jul 31 18:43:44.384333 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.384304    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6435d2b7_3eb8_4ee0_ab81_6c7862a7d376.slice/crio-a231f807dd11e481b6109024d72840b90fefa0b0a5b86ca662a84bb974e1b74a WatchSource:0}: Error finding container a231f807dd11e481b6109024d72840b90fefa0b0a5b86ca662a84bb974e1b74a: Status 404 returned error can't find the container with id a231f807dd11e481b6109024d72840b90fefa0b0a5b86ca662a84bb974e1b74a
Jul 31 18:43:44.386645 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.386626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx"
Jul 31 18:43:44.395921 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.395897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rprxx"
Jul 31 18:43:44.398016 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.397993    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467a45a8_1a97_4162_8310_40ab1a20a1a5.slice/crio-1635b2413ebfe830b93a6e5c94599c16568c7dffbca61d95d36f9b9b671bc05c WatchSource:0}: Error finding container 1635b2413ebfe830b93a6e5c94599c16568c7dffbca61d95d36f9b9b671bc05c: Status 404 returned error can't find the container with id 1635b2413ebfe830b93a6e5c94599c16568c7dffbca61d95d36f9b9b671bc05c
Jul 31 18:43:44.407138 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.407092    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ed081d_85b1_41da_af82_2caff6b85486.slice/crio-aa21e3dbf0507d9960aa0595e6f8e6d5ebdab32f796f513365192c2ecbf969fb WatchSource:0}: Error finding container aa21e3dbf0507d9960aa0595e6f8e6d5ebdab32f796f513365192c2ecbf969fb: Status 404 returned error can't find the container with id aa21e3dbf0507d9960aa0595e6f8e6d5ebdab32f796f513365192c2ecbf969fb
Jul 31 18:43:44.407912 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.407754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d66d8"
Jul 31 18:43:44.415261 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.415237    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vcn4"
Jul 31 18:43:44.425064 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.424874    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-68swk"
Jul 31 18:43:44.425227 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.425177    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06af2d8b_c23f_4b50_9cc4_5da5cc5b3880.slice/crio-dd6f9db1112bd885b8a1217093d2c92d7a0c1baea64f5454cb5b8c2589489239 WatchSource:0}: Error finding container dd6f9db1112bd885b8a1217093d2c92d7a0c1baea64f5454cb5b8c2589489239: Status 404 returned error can't find the container with id dd6f9db1112bd885b8a1217093d2c92d7a0c1baea64f5454cb5b8c2589489239
Jul 31 18:43:44.434297 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.434272    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:43:44.436962 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.436925    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4505fee6_5b31_49a9_8125_2c57e9ae7be4.slice/crio-3b57fbf3c72911d5a28f4f8dc956ba9457e289ba2f513d98f82cbcd2b95c8819 WatchSource:0}: Error finding container 3b57fbf3c72911d5a28f4f8dc956ba9457e289ba2f513d98f82cbcd2b95c8819: Status 404 returned error can't find the container with id 3b57fbf3c72911d5a28f4f8dc956ba9457e289ba2f513d98f82cbcd2b95c8819
Jul 31 18:43:44.443445 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.443266    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgsr2"
Jul 31 18:43:44.446013 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.445986    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13caa362_adf4_45d4_a4da_fadf86931e04.slice/crio-82ce29b3f1d3c571379f3086133ca5f15d9e08cac862ff3c1b7683f8e2a82d12 WatchSource:0}: Error finding container 82ce29b3f1d3c571379f3086133ca5f15d9e08cac862ff3c1b7683f8e2a82d12: Status 404 returned error can't find the container with id 82ce29b3f1d3c571379f3086133ca5f15d9e08cac862ff3c1b7683f8e2a82d12
Jul 31 18:43:44.449956 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.449930    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:43:44.454945 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.454918    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4598ba_52ca_438c_bfff_f680e640210d.slice/crio-91d12204b97a94535976557d588b0ca9f8d92ac76a8bbad4be4eaa7f242f27e4 WatchSource:0}: Error finding container 91d12204b97a94535976557d588b0ca9f8d92ac76a8bbad4be4eaa7f242f27e4: Status 404 returned error can't find the container with id 91d12204b97a94535976557d588b0ca9f8d92ac76a8bbad4be4eaa7f242f27e4
Jul 31 18:43:44.461410 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:43:44.461388    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e00fd6_040c_4723_b241_289e17f0ef8e.slice/crio-1febb38b82a72879d16bbdeb0a41ec25d5470501c098b981ebbcfdd3971b6e78 WatchSource:0}: Error finding container 1febb38b82a72879d16bbdeb0a41ec25d5470501c098b981ebbcfdd3971b6e78: Status 404 returned error can't find the container with id 1febb38b82a72879d16bbdeb0a41ec25d5470501c098b981ebbcfdd3971b6e78
Jul 31 18:43:44.685409 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.685327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:44.685592 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.685465    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:44.685592 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.685553    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:43:45.685531646 +0000 UTC m=+4.033113510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:44.786144 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.786104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:44.786335 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.786314    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:44.786423 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.786402    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:45.786380692 +0000 UTC m=+4.133962575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:44.886922 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:44.886886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:44.887108 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.887036    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:43:44.887108 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.887059    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:43:44.887108 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.887070    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:44.887209 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:44.887127    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:45.887113349 +0000 UTC m=+4.234695209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:45.125509 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.125410    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:38:44 +0000 UTC" deadline="2028-04-23 03:01:28.828049538 +0000 UTC"
Jul 31 18:43:45.125509 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.125485    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15152h17m43.70257035s"
Jul 31 18:43:45.178110 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.178046    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:45.178318 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.178241    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:45.187991 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.187952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-99v4g" event={"ID":"89e00fd6-040c-4723-b241-289e17f0ef8e","Type":"ContainerStarted","Data":"1febb38b82a72879d16bbdeb0a41ec25d5470501c098b981ebbcfdd3971b6e78"}
Jul 31 18:43:45.189839 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.189789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerStarted","Data":"3b57fbf3c72911d5a28f4f8dc956ba9457e289ba2f513d98f82cbcd2b95c8819"}
Jul 31 18:43:45.191116 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.191081    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vcn4" event={"ID":"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880","Type":"ContainerStarted","Data":"dd6f9db1112bd885b8a1217093d2c92d7a0c1baea64f5454cb5b8c2589489239"}
Jul 31 18:43:45.192532 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.192506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d66d8" event={"ID":"9f8de46e-3b7b-4896-a7c4-2256edda4630","Type":"ContainerStarted","Data":"f2a06befc5d152ba823970fdac39944371244755b4593e6c9f1f1933f9a7135c"}
Jul 31 18:43:45.193914 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.193890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx" event={"ID":"467a45a8-1a97-4162-8310-40ab1a20a1a5","Type":"ContainerStarted","Data":"1635b2413ebfe830b93a6e5c94599c16568c7dffbca61d95d36f9b9b671bc05c"}
Jul 31 18:43:45.195276 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.195253    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgsr2" event={"ID":"bb4598ba-52ca-438c-bfff-f680e640210d","Type":"ContainerStarted","Data":"91d12204b97a94535976557d588b0ca9f8d92ac76a8bbad4be4eaa7f242f27e4"}
Jul 31 18:43:45.196771 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.196741    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"82ce29b3f1d3c571379f3086133ca5f15d9e08cac862ff3c1b7683f8e2a82d12"}
Jul 31 18:43:45.198537 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.198333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rprxx" event={"ID":"f7ed081d-85b1-41da-af82-2caff6b85486","Type":"ContainerStarted","Data":"aa21e3dbf0507d9960aa0595e6f8e6d5ebdab32f796f513365192c2ecbf969fb"}
Jul 31 18:43:45.199608 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.199583    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7tgm2" event={"ID":"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376","Type":"ContainerStarted","Data":"a231f807dd11e481b6109024d72840b90fefa0b0a5b86ca662a84bb974e1b74a"}
Jul 31 18:43:45.694275 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.694226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:45.694507 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.694402    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:45.694507 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.694471    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.69445094 +0000 UTC m=+6.042032815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:45.795684 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.795656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:45.795898 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.795881    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:45.795982 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.795943    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.795928487 +0000 UTC m=+6.143510347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:45.897168 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:45.896197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:45.897168 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.896532    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:43:45.897168 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.896555    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:43:45.897168 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.896568    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:45.897168 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:45.896617    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:47.896602157 +0000 UTC m=+6.244184031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:46.126058 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.126016    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 18:38:44 +0000 UTC" deadline="2028-05-03 16:58:14.126618994 +0000 UTC"
Jul 31 18:43:46.126058 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.126057    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15406h14m28.000567457s"
Jul 31 18:43:46.184557 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.184481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:46.184775 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.184481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:46.184970 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:46.184928    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:46.185043 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:46.184746    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:46.209546 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.208748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" event={"ID":"fda49057b793c5c62334b176ef9bc360","Type":"ContainerStarted","Data":"386ccb43165ac2ad390b327a248b33ac3355955106601aa9ed89c5f1cda5423b"}
Jul 31 18:43:46.229923 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:46.229470    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" podStartSLOduration=2.22945041 podStartE2EDuration="2.22945041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:43:46.228387246 +0000 UTC m=+4.575969131" watchObservedRunningTime="2026-07-31 18:43:46.22945041 +0000 UTC m=+4.577032298"
Jul 31 18:43:47.178307 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.178268    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:47.178846 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.178435    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:47.235717 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.235655    2557 generic.go:356] "Generic (PLEG): container finished" podID="9d6059c7ee63b4af2d85012a3466eef7" containerID="3bfed1d13b958509cab36147cc7e1956f5fbf6a6c5316c33e289a9e3d0b3abf5" exitCode=0
Jul 31 18:43:47.235899 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.235828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"9d6059c7ee63b4af2d85012a3466eef7","Type":"ContainerDied","Data":"3bfed1d13b958509cab36147cc7e1956f5fbf6a6c5316c33e289a9e3d0b3abf5"}
Jul 31 18:43:47.713733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.713014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:47.713733 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.713223    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:47.713733 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.713293    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:43:51.713273209 +0000 UTC m=+10.060855085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:47.814495 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.813810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:47.814495 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.814058    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:47.814495 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.814128    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:51.814105599 +0000 UTC m=+10.161687463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:47.914985 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:47.914942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:47.915191 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.915156    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:43:47.915191 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.915180    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:43:47.915314 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.915194    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:47.915314 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:47.915258    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:51.915239132 +0000 UTC m=+10.262820994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:48.180376 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:48.180289    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:48.180868 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:48.180440    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:48.180955 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:48.180874    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:48.181014 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:48.180973    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:48.254193 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:48.251197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"9d6059c7ee63b4af2d85012a3466eef7","Type":"ContainerStarted","Data":"b435b98db858c0bfeb94d0eb764dad0aa5761e698a0f2fb33a96de102f69c20c"}
Jul 31 18:43:49.178318 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:49.178276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:49.178533 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:49.178492    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:50.181795 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:50.181003    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:50.181795 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:50.181139    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:50.181795 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:50.181589    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:50.181795 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:50.181704    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:51.178260 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:51.178220    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:51.178478 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.178360    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:51.748790 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:51.748693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:51.749203 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.748886    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:51.749203 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.748975    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:43:59.748952111 +0000 UTC m=+18.096533973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:51.849536 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:51.849412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:51.849710 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.849596    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:51.849710 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.849683    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:59.849660705 +0000 UTC m=+18.197242582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:51.950223 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:51.950182    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:51.950452 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.950408    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:43:51.950452 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.950443    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:43:51.950603 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.950541    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:51.950659 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:51.950616    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:43:59.950599762 +0000 UTC m=+18.298181636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:43:52.178617 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:52.178518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:52.178783 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:52.178653    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:52.179064 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:52.179039    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:52.179171 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:52.179137    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:53.178342 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:53.178302    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:53.178900 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:53.178492    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:54.179158 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:54.178617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:54.179158 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:54.178777    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:54.179158 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:54.178787    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:54.179158 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:54.178919    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:55.177499 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:55.177463    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:55.177681 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:55.177587    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:56.179119 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:56.179089    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:56.179567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:56.179205    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:56.179567 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:56.179089    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:56.179645 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:56.179610    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:57.178260 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:57.178213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:57.178458 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:57.178370    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:58.177724 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:58.177691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:58.178149 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:58.177834    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:43:58.178149 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:58.177887    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:43:58.178149 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:58.178014    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:43:59.178323 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:59.178288    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:59.178810 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:59.178409    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:43:59.816185 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:59.816130    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:43:59.816428 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:59.816301    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:59.816428 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:59.816375    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:15.816357532 +0000 UTC m=+34.163939393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:43:59.916564 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:43:59.916526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:43:59.916764 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:59.916704    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:43:59.916831 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:43:59.916784    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:15.916765742 +0000 UTC m=+34.264347599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:44:00.017930 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:00.017892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:00.018174 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.018040    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:44:00.018174 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.018063    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:44:00.018174 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.018075    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:44:00.018174 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.018131    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:16.018113892 +0000 UTC m=+34.365695770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:44:00.178252 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:00.178158    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:00.178421 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:00.178165    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:00.178421 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.178287    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:00.178421 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:00.178358    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:01.177847 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:01.177820    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:01.178014 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:01.177955    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:02.178730 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:02.178688    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:02.179262 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:02.178790    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:02.179262 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:02.178885    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:02.179262 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:02.178979    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:03.178164 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:03.178129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:03.178339 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:03.178264    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:04.177605 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:04.177571    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:04.178066 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:04.177709    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:04.178066 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:04.177764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:04.178066 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:04.177853    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:05.177732 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.177407    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:05.178346 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:05.177828    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:05.291313 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.291279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vcn4" event={"ID":"06af2d8b-c23f-4b50-9cc4-5da5cc5b3880","Type":"ContainerStarted","Data":"e851b42d77cdfc1856f2205d779c9af04b8e66bc2a0333bc592b0ec9f50f27f5"}
Jul 31 18:44:05.292705 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.292682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d66d8" event={"ID":"9f8de46e-3b7b-4896-a7c4-2256edda4630","Type":"ContainerStarted","Data":"faea3875e1511549e5593c0c158bd6972c3929b296480ecbc744083aaefa82ab"}
Jul 31 18:44:05.294261 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.294237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx" event={"ID":"467a45a8-1a97-4162-8310-40ab1a20a1a5","Type":"ContainerStarted","Data":"a2ac29d14ee3719b2cb60340be958060184a427b6eb0e7235ddfc9b74ce58a19"}
Jul 31 18:44:05.297049 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"33aa84106d3ff3fd1358979253bf8a12d0d869f7416b9e5ffff058fdec9b360f"}
Jul 31 18:44:05.297141 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"dafa4ebabfed04fe8c905ecb5eadb46b8c73c2483c5545d5273e0d6f3b9c9d9d"}
Jul 31 18:44:05.297141 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"c59086d86e5d1135b97092bd165470fa2893dcb7ab084b8a114a80bc86634efa"}
Jul 31 18:44:05.297271 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297199    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"288b970713a76011d6ace6b0009fc9c3ba93701eb756e5b0934710e446b88117"}
Jul 31 18:44:05.297271 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"1030fe1620b7cf11647a1bba8f687e854c7957c657b91f67071ebf99ae620b3b"}
Jul 31 18:44:05.297364 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.297294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"8d3b961cc508e9563aba790f32223a46e1ae15d84d20ae4e2bb50e25dccdf6c4"}
Jul 31 18:44:05.298411 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.298389    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rprxx" event={"ID":"f7ed081d-85b1-41da-af82-2caff6b85486","Type":"ContainerStarted","Data":"250b8c0a0ad55da847fee633e108ee806f2b01e836752fc2e884d1aab005c483"}
Jul 31 18:44:05.299813 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.299793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7tgm2" event={"ID":"6435d2b7-3eb8-4ee0-ab81-6c7862a7d376","Type":"ContainerStarted","Data":"9349c804a481ba5f77fb166f8d9f7c41064fc7c0c9451d2ef7c5de8e8b4fbafa"}
Jul 31 18:44:05.301210 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.301177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-99v4g" event={"ID":"89e00fd6-040c-4723-b241-289e17f0ef8e","Type":"ContainerStarted","Data":"04ed3b5cff7ebbb40deffc26fe0ae2353bd54dbfc09c4c96b213f811448b6b6d"}
Jul 31 18:44:05.302653 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.302630    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="41567d3fd807304f8a1a74aaadbb01ba6e42b277c6c05e966cf80a6722a7c365" exitCode=0
Jul 31 18:44:05.302731 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.302663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"41567d3fd807304f8a1a74aaadbb01ba6e42b277c6c05e966cf80a6722a7c365"}
Jul 31 18:44:05.313622 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.313584    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4vcn4" podStartSLOduration=2.080528126 podStartE2EDuration="22.313571138s" totalImagesPullingTime="20.233043012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:05.313078179 +0000 UTC m=+23.660660059" watchObservedRunningTime="2026-07-31 18:44:05.313571138 +0000 UTC m=+23.661153020"
Jul 31 18:44:05.313949 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.313927    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" podStartSLOduration=21.313919702 podStartE2EDuration="21.313919702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:43:48.280651048 +0000 UTC m=+6.628232936" watchObservedRunningTime="2026-07-31 18:44:05.313919702 +0000 UTC m=+23.661501581"
Jul 31 18:44:05.329319 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.329272    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rprxx" podStartSLOduration=2.132619811 podStartE2EDuration="22.329258664s" totalImagesPullingTime="20.196638853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:05.329006811 +0000 UTC m=+23.676588692" watchObservedRunningTime="2026-07-31 18:44:05.329258664 +0000 UTC m=+23.676840543"
Jul 31 18:44:05.345714 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.345669    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d66d8" podStartSLOduration=2.178859147 podStartE2EDuration="22.34565552s" totalImagesPullingTime="20.166796373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:05.3452359 +0000 UTC m=+23.692817779" watchObservedRunningTime="2026-07-31 18:44:05.34565552 +0000 UTC m=+23.693237400"
Jul 31 18:44:05.386601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:05.386513    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-99v4g" podStartSLOduration=6.174205723 podStartE2EDuration="22.38649912s" totalImagesPullingTime="16.212293397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:05.386327686 +0000 UTC m=+23.733909566" watchObservedRunningTime="2026-07-31 18:44:05.38649912 +0000 UTC m=+23.734081000"
Jul 31 18:44:06.177858 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.177784    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:06.178305 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:06.177913    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:06.178305 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.177995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:06.178305 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:06.178075    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:06.306061 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.306034    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgsr2" event={"ID":"bb4598ba-52ca-438c-bfff-f680e640210d","Type":"ContainerStarted","Data":"adc526ec9325397e852bd1f625ce5c0a9d29ec0070482040f2832ad504b428c9"}
Jul 31 18:44:06.325661 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.325609    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7tgm2" podStartSLOduration=3.108858682 podStartE2EDuration="23.325587479s" totalImagesPullingTime="20.216728797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:05.404790298 +0000 UTC m=+23.752372177" watchObservedRunningTime="2026-07-31 18:44:06.325587479 +0000 UTC m=+24.673169360"
Jul 31 18:44:06.325979 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.325939    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hgsr2" podStartSLOduration=3.178110159 podStartE2EDuration="23.325929385s" totalImagesPullingTime="20.147819226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:06.325715185 +0000 UTC m=+24.673297056" watchObservedRunningTime="2026-07-31 18:44:06.325929385 +0000 UTC m=+24.673511269"
Jul 31 18:44:06.414892 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:06.414863    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 18:44:07.135459 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:07.135318    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T18:44:06.414883661Z","UUID":"85188d7f-9f92-47d2-9efd-4e2fe683d1fa","Handler":null,"Name":"","Endpoint":""}
Jul 31 18:44:07.137093 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:07.137063    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 18:44:07.137230 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:07.137112    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 18:44:07.177902 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:07.177876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:07.178380 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:07.178019    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:07.311876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:07.311627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx" event={"ID":"467a45a8-1a97-4162-8310-40ab1a20a1a5","Type":"ContainerStarted","Data":"b9b79787d56b0f5c8093b192858c826cdae3d8a3dee8dd048a7d589b97caa645"}
Jul 31 18:44:08.178215 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:08.178134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:08.178617 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:08.178238    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:08.178617 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:08.178293    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:08.178617 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:08.178399    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:08.316636 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:08.316598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx" event={"ID":"467a45a8-1a97-4162-8310-40ab1a20a1a5","Type":"ContainerStarted","Data":"b85ba4f7a68e94224455ee6ab9c2942f4f9135cdec5af592bb735f91eb330b28"}
Jul 31 18:44:08.320891 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:08.320858    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"3f597875e1544976fc2a3dfc23680a6a380d6b82c5438cd228f40c3d550504f0"}
Jul 31 18:44:08.337799 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:08.337747    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2qjx" podStartSLOduration=2.365950235 podStartE2EDuration="25.337728456s" totalImagesPullingTime="22.971778221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:08.33711448 +0000 UTC m=+26.684696361" watchObservedRunningTime="2026-07-31 18:44:08.337728456 +0000 UTC m=+26.685310337"
Jul 31 18:44:09.178609 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:09.178360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:09.179034 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:09.178729    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:09.746207 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:09.746171    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:44:09.746896 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:09.746866    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:44:10.181462 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:10.181408    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:10.181913 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:10.181554    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:10.181913 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:10.181608    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:10.181913 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:10.181708    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:10.325736 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:10.325702    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:44:10.326179 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:10.326155    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-99v4g"
Jul 31 18:44:11.178255 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:11.178234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:11.178365 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:11.178348    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:11.330948 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:11.330757    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" event={"ID":"13caa362-adf4-45d4-a4da-fadf86931e04","Type":"ContainerStarted","Data":"02920a225c4b7955705eae4bffaa9d2b45313dc048c3ade99a2a0c86f7ce243b"}
Jul 31 18:44:11.367868 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:11.367810    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9" podStartSLOduration=8.134743224 podStartE2EDuration="28.367788264s" totalImagesPullingTime="20.23304504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:11.36754465 +0000 UTC m=+29.715126530" watchObservedRunningTime="2026-07-31 18:44:11.367788264 +0000 UTC m=+29.715370145"
Jul 31 18:44:12.178617 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.178583    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:12.178773 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:12.178658    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:12.178773 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.178740    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:12.178887 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:12.178865    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:12.335300 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.335260    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="b574644214fd7cc46ca0c89a3e247019432d7f28db0bea5c1ed308a74232a6c2" exitCode=0
Jul 31 18:44:12.335886 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.335356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"b574644214fd7cc46ca0c89a3e247019432d7f28db0bea5c1ed308a74232a6c2"}
Jul 31 18:44:12.336070 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.336051    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:12.336149 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.336126    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:12.336195 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.336177    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:12.361153 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.361126    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:12.362538 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:12.362518    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:13.178410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.178373    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:13.178605 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:13.178510    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:13.538142 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.537240    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stg44"]
Jul 31 18:44:13.538142 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.537374    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:13.538142 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:13.537521    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:13.538802 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.538720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhf8w"]
Jul 31 18:44:13.538875 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.538812    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:13.538936 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:13.538897    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:13.540031 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.540007    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gnwc"]
Jul 31 18:44:13.540144 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:13.540098    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:13.540204 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:13.540185    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:14.342345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:14.342107    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="cf66407b0e112b25100824d574e92f58313d92cb20f29a396ddc581402965417" exitCode=0
Jul 31 18:44:14.342345 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:14.342201    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"cf66407b0e112b25100824d574e92f58313d92cb20f29a396ddc581402965417"}
Jul 31 18:44:15.177794 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:15.177764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:15.178179 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:15.177767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:15.178179 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:15.177903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:15.178179 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.177869    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:15.178179 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.177994    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:15.178179 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.178042    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:15.831794 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:15.831694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:15.831946 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.831844    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:44:15.831946 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.831910    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:47.831894307 +0000 UTC m=+66.179476165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 18:44:15.932208 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:15.932169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:15.932358 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.932320    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 18:44:15.932421 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:15.932381    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret podName:df4356d7-969b-4b93-a3e6-3fb45690cd29 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:47.932366743 +0000 UTC m=+66.279948602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret") pod "global-pull-secret-syncer-fhf8w" (UID: "df4356d7-969b-4b93-a3e6-3fb45690cd29") : object "kube-system"/"original-pull-secret" not registered
Jul 31 18:44:16.033057 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:16.033022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:16.033219 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:16.033155    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 18:44:16.033219 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:16.033171    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 18:44:16.033219 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:16.033179    2557 projected.go:196] Error preparing data for projected volume kube-api-access-4pdlj for pod openshift-network-diagnostics/network-check-target-5gnwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:44:16.033336 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:16.033226    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj podName:2ce52429-d43f-4e1c-b2d1-2825b30277b7 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:48.033214553 +0000 UTC m=+66.380796425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pdlj" (UniqueName: "kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj") pod "network-check-target-5gnwc" (UID: "2ce52429-d43f-4e1c-b2d1-2825b30277b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 18:44:16.348737 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:16.348702    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="a3e63aaf7901cd718aa75438c745c3b2dd4e66913ac9154f597201ea6d8aa0bb" exitCode=0
Jul 31 18:44:16.349195 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:16.348739    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"a3e63aaf7901cd718aa75438c745c3b2dd4e66913ac9154f597201ea6d8aa0bb"}
Jul 31 18:44:17.178017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.177971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:17.178017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.177993    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:17.178207 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.177971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:17.178207 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.178129    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-stg44" podUID="53ebbbe8-1bcd-49e4-9f36-f95c74d6534b"
Jul 31 18:44:17.178286 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.178230    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhf8w" podUID="df4356d7-969b-4b93-a3e6-3fb45690cd29"
Jul 31 18:44:17.178343 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.178313    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gnwc" podUID="2ce52429-d43f-4e1c-b2d1-2825b30277b7"
Jul 31 18:44:17.314101 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.314067    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeReady"
Jul 31 18:44:17.314273 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.314240    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 18:44:17.357908 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.357871    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:44:17.361827 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.361805    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.364876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.364857    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 18:44:17.364972 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.364857    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4pbfx\""
Jul 31 18:44:17.364972 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.364857    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 18:44:17.365214 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.365199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 18:44:17.372405 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.372388    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 18:44:17.373124 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.373105    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:44:17.373822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.373803    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5ps6l"]
Jul 31 18:44:17.377680 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.377629    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvps"]
Jul 31 18:44:17.377768 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.377742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.380428 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.380409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 18:44:17.380512 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.380495    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 18:44:17.380601 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.380586    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8wjd\""
Jul 31 18:44:17.381115 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.381100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.383761 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.383526    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 18:44:17.383761 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.383537    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96c4f\""
Jul 31 18:44:17.383761 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.383605    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 18:44:17.383761 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.383655    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 18:44:17.389997 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.389971    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ps6l"]
Jul 31 18:44:17.390847 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.390776    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvps"]
Jul 31 18:44:17.443646 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.443822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ceefe50-afa0-49e4-ab29-a898d0173807-cert\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.443822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0497e7-04d8-4120-81fe-23357e4d9b1b-config-volume\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.443822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.443822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n797\" (UniqueName: \"kubernetes.io/projected/5a0497e7-04d8-4120-81fe-23357e4d9b1b-kube-api-access-2n797\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.443822 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh54\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443932    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.443975    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.444022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.444012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbcp\" (UniqueName: \"kubernetes.io/projected/8ceefe50-afa0-49e4-ab29-a898d0173807-kube-api-access-lbbcp\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.444257 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.444034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0497e7-04d8-4120-81fe-23357e4d9b1b-tmp-dir\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.444257 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.444058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545114 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545304 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545304 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545168    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545304 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545304 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545610 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545563    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbcp\" (UniqueName: \"kubernetes.io/projected/8ceefe50-afa0-49e4-ab29-a898d0173807-kube-api-access-lbbcp\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.545610 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0497e7-04d8-4120-81fe-23357e4d9b1b-tmp-dir\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ceefe50-afa0-49e4-ab29-a898d0173807-cert\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545739    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0497e7-04d8-4120-81fe-23357e4d9b1b-config-volume\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545747    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.545769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.546096 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n797\" (UniqueName: \"kubernetes.io/projected/5a0497e7-04d8-4120-81fe-23357e4d9b1b-kube-api-access-2n797\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.546096 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.545826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh54\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.546096 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.545839    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:17.546096 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.545853    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:17.546096 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.545904    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:18.045886621 +0000 UTC m=+36.393468493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:17.547085 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.546613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0497e7-04d8-4120-81fe-23357e4d9b1b-config-volume\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.547085 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.546709    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:17.547085 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:17.546758    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:18.046741528 +0000 UTC m=+36.394323392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:17.547085 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.546985    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.547085 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.547040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0497e7-04d8-4120-81fe-23357e4d9b1b-tmp-dir\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.550228 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.550208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.550333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.550245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.550333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.550278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ceefe50-afa0-49e4-ab29-a898d0173807-cert\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.555592 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.555568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.555850 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.555820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbcp\" (UniqueName: \"kubernetes.io/projected/8ceefe50-afa0-49e4-ab29-a898d0173807-kube-api-access-lbbcp\") pod \"ingress-canary-2nvps\" (UID: \"8ceefe50-afa0-49e4-ab29-a898d0173807\") " pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.555930 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.555820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh54\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:17.556545 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.556525    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n797\" (UniqueName: \"kubernetes.io/projected/5a0497e7-04d8-4120-81fe-23357e4d9b1b-kube-api-access-2n797\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:17.697043 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.696961    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2nvps"
Jul 31 18:44:17.852053 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:17.851962    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2nvps"]
Jul 31 18:44:17.855286 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:17.855262    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ceefe50_afa0_49e4_ab29_a898d0173807.slice/crio-0eb19ce06f33b98eeeccc231a74c8843a904bcde8d1e491de33e710d0bd93f4d WatchSource:0}: Error finding container 0eb19ce06f33b98eeeccc231a74c8843a904bcde8d1e491de33e710d0bd93f4d: Status 404 returned error can't find the container with id 0eb19ce06f33b98eeeccc231a74c8843a904bcde8d1e491de33e710d0bd93f4d
Jul 31 18:44:18.049970 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:18.049673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:18.050123 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:18.049847    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:18.050123 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:18.050021    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:18.050123 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:18.050015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:18.050123 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:18.050087    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:19.050070661 +0000 UTC m=+37.397652541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:18.050123 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:18.050102    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:18.050361 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:18.050159    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:19.050142358 +0000 UTC m=+37.397724222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:18.354857 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:18.354822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2nvps" event={"ID":"8ceefe50-afa0-49e4-ab29-a898d0173807","Type":"ContainerStarted","Data":"0eb19ce06f33b98eeeccc231a74c8843a904bcde8d1e491de33e710d0bd93f4d"}
Jul 31 18:44:19.059256 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.059215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.059288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:19.059383    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:19.059394    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:19.059401    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:19.059473    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:21.059436645 +0000 UTC m=+39.407018517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:19.059979 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:19.059487    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:21.05948119 +0000 UTC m=+39.407063050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:19.178051 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.178016    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:19.178432 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.178414    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:19.179002 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.178668    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:19.184633 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.184612    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 18:44:19.184738 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.184719    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 18:44:19.184885 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.184865    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 18:44:19.185821 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.185800    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\""
Jul 31 18:44:19.186067 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.186051    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\""
Jul 31 18:44:19.186170 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:19.186056    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 18:44:21.078055 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:21.078010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:21.078115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:21.078179    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:21.078249    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:21.078263    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:21.078265    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:25.078246042 +0000 UTC m=+43.425827905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:21.078567 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:21.078323    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:25.078307459 +0000 UTC m=+43.425889326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:22.366105 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:22.366076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2nvps" event={"ID":"8ceefe50-afa0-49e4-ab29-a898d0173807","Type":"ContainerStarted","Data":"130641edfaa86902a9f1018aae57f537067433816f040d8c5a6d200cf709943b"}
Jul 31 18:44:22.368516 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:22.368493    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="dd2c2b9ee5c800f1454704ceafe5d32b5e8ef2a55ca3b9ba17d2b2f7df94ad49" exitCode=0
Jul 31 18:44:22.368636 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:22.368523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"dd2c2b9ee5c800f1454704ceafe5d32b5e8ef2a55ca3b9ba17d2b2f7df94ad49"}
Jul 31 18:44:22.389856 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:22.389814    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2nvps" podStartSLOduration=1.338338701 podStartE2EDuration="5.389799484s" totalImagesPullingTime="4.051460783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:22.389531907 +0000 UTC m=+40.737113787" watchObservedRunningTime="2026-07-31 18:44:22.389799484 +0000 UTC m=+40.737381382"
Jul 31 18:44:23.373676 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:23.373643    2557 generic.go:356] "Generic (PLEG): container finished" podID="4505fee6-5b31-49a9-8125-2c57e9ae7be4" containerID="c37b2f55a2a15e36cc1b52a8faec0c6edc302939ecbc4720689b3b744d3abbf4" exitCode=0
Jul 31 18:44:23.374016 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:23.373724    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerDied","Data":"c37b2f55a2a15e36cc1b52a8faec0c6edc302939ecbc4720689b3b744d3abbf4"}
Jul 31 18:44:24.378599 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:24.378564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-68swk" event={"ID":"4505fee6-5b31-49a9-8125-2c57e9ae7be4","Type":"ContainerStarted","Data":"0b959d5e62fcc97d6c3b4c268c758d2d22714f764e7b56cb9e73c38ddbf5fe5f"}
Jul 31 18:44:24.411471 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:24.411409    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-68swk" podStartSLOduration=7.419792829 podStartE2EDuration="41.411389874s" totalImagesPullingTime="33.991597045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:24.411094974 +0000 UTC m=+42.758676865" watchObservedRunningTime="2026-07-31 18:44:24.411389874 +0000 UTC m=+42.758971753"
Jul 31 18:44:25.108843 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:25.108804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:25.109019 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:25.108869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:25.109019 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:25.108973    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:25.109019 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:25.108981    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:25.109019 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:25.109003    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:25.109227 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:25.109027    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.109012793 +0000 UTC m=+51.456594651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:25.109227 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:25.109054    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:33.109037691 +0000 UTC m=+51.456619562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:32.803024 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.802990    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"]
Jul 31 18:44:32.815403 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.815376    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"]
Jul 31 18:44:32.815578 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.815513    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"
Jul 31 18:44:32.819043 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.819021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j2p57\""
Jul 31 18:44:32.829487 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.829465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"]
Jul 31 18:44:32.829607 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.829508    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:32.829607 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.829533    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"]
Jul 31 18:44:32.832322 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.832305    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 18:44:32.832953 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.832938    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 18:44:32.833092 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.833074    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4rbql\""
Jul 31 18:44:32.834551 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.834530    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:44:32.836655 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.836634    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 18:44:32.888208 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.888175    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"]
Jul 31 18:44:32.911492 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.911433    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b2gjc"]
Jul 31 18:44:32.911667 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.911574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"
Jul 31 18:44:32.916004 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.915975    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ds4sc\""
Jul 31 18:44:32.916358 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.916339    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:44:32.916416 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.916373    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 18:44:32.928017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.927991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"]
Jul 31 18:44:32.928142 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.928033    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b2gjc"]
Jul 31 18:44:32.928142 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.928096    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:32.931132 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.931112    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xpd9d\""
Jul 31 18:44:32.934542 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.934378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 18:44:32.934542 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.934495    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:44:32.934677 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.934642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 18:44:32.934677 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.934646    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 18:44:32.941487 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.941469    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 18:44:32.972188 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.972163    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkj6c\" (UniqueName: \"kubernetes.io/projected/62dedd37-4d36-4735-b54f-1206da18cb5f-kube-api-access-lkj6c\") pod \"network-check-source-65bdbbff45-4l8h2\" (UID: \"62dedd37-4d36-4735-b54f-1206da18cb5f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"
Jul 31 18:44:32.972277 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.972208    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0353f16-2658-49f0-b4de-87d3bcf0d529-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:32.972277 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.972231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0353f16-2658-49f0-b4de-87d3bcf0d529-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:32.972277 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:32.972247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjbq\" (UniqueName: \"kubernetes.io/projected/d0353f16-2658-49f0-b4de-87d3bcf0d529-kube-api-access-wkjbq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.072944 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072870    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0353f16-2658-49f0-b4de-87d3bcf0d529-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.072944 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072902    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0353f16-2658-49f0-b4de-87d3bcf0d529-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.072944 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjbq\" (UniqueName: \"kubernetes.io/projected/d0353f16-2658-49f0-b4de-87d3bcf0d529-kube-api-access-wkjbq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.072944 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8725d\" (UniqueName: \"kubernetes.io/projected/7031db7a-a3ee-45f4-8448-4913e8a110c9-kube-api-access-8725d\") pod \"volume-data-source-validator-5b8776669-g5mf5\" (UID: \"7031db7a-a3ee-45f4-8448-4913e8a110c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"
Jul 31 18:44:33.073259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-config\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.073259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.072983    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-trusted-ca\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.073259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.073124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0140a-41df-45f3-bbd3-0cc2d319a729-serving-cert\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.073259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.073185    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkj6c\" (UniqueName: \"kubernetes.io/projected/62dedd37-4d36-4735-b54f-1206da18cb5f-kube-api-access-lkj6c\") pod \"network-check-source-65bdbbff45-4l8h2\" (UID: \"62dedd37-4d36-4735-b54f-1206da18cb5f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"
Jul 31 18:44:33.073259 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.073214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66l55\" (UniqueName: \"kubernetes.io/projected/7da0140a-41df-45f3-bbd3-0cc2d319a729-kube-api-access-66l55\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.073536 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.073520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0353f16-2658-49f0-b4de-87d3bcf0d529-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.076272 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.076255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0353f16-2658-49f0-b4de-87d3bcf0d529-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.083940 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.083918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjbq\" (UniqueName: \"kubernetes.io/projected/d0353f16-2658-49f0-b4de-87d3bcf0d529-kube-api-access-wkjbq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-lqgxh\" (UID: \"d0353f16-2658-49f0-b4de-87d3bcf0d529\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.084096 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.084078    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkj6c\" (UniqueName: \"kubernetes.io/projected/62dedd37-4d36-4735-b54f-1206da18cb5f-kube-api-access-lkj6c\") pod \"network-check-source-65bdbbff45-4l8h2\" (UID: \"62dedd37-4d36-4735-b54f-1206da18cb5f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"
Jul 31 18:44:33.124082 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.124043    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"
Jul 31 18:44:33.138207 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.138187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"
Jul 31 18:44:33.173604 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8725d\" (UniqueName: \"kubernetes.io/projected/7031db7a-a3ee-45f4-8448-4913e8a110c9-kube-api-access-8725d\") pod \"volume-data-source-validator-5b8776669-g5mf5\" (UID: \"7031db7a-a3ee-45f4-8448-4913e8a110c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"
Jul 31 18:44:33.173733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-config\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.173733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-trusted-ca\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.173733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:33.173733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0140a-41df-45f3-bbd3-0cc2d319a729-serving-cert\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.173938 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:33.173838    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:33.173938 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:33.173860    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:33.173938 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173898    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66l55\" (UniqueName: \"kubernetes.io/projected/7da0140a-41df-45f3-bbd3-0cc2d319a729-kube-api-access-66l55\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.173938 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:33.173935    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:44:49.173912894 +0000 UTC m=+67.521494777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:33.174151 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.173990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:33.174151 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:33.174100    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:33.174151 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:33.174140    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:44:49.174127316 +0000 UTC m=+67.521709195 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:33.174477 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.174426    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-config\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.174905 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.174827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0140a-41df-45f3-bbd3-0cc2d319a729-trusted-ca\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.176194 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.176177    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0140a-41df-45f3-bbd3-0cc2d319a729-serving-cert\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.185289 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.185259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8725d\" (UniqueName: \"kubernetes.io/projected/7031db7a-a3ee-45f4-8448-4913e8a110c9-kube-api-access-8725d\") pod \"volume-data-source-validator-5b8776669-g5mf5\" (UID: \"7031db7a-a3ee-45f4-8448-4913e8a110c9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"
Jul 31 18:44:33.185546 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.185528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66l55\" (UniqueName: \"kubernetes.io/projected/7da0140a-41df-45f3-bbd3-0cc2d319a729-kube-api-access-66l55\") pod \"console-operator-5ccd6b445f-b2gjc\" (UID: \"7da0140a-41df-45f3-bbd3-0cc2d319a729\") " pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.221473 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.220964    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"
Jul 31 18:44:33.237300 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.237273    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:33.283544 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.283323    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2"]
Jul 31 18:44:33.287687 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:33.287655    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dedd37_4d36_4735_b54f_1206da18cb5f.slice/crio-030c852650ddeaabb0ec6eb6e6763877ed0e82dabb34f0716e50c144c643db00 WatchSource:0}: Error finding container 030c852650ddeaabb0ec6eb6e6763877ed0e82dabb34f0716e50c144c643db00: Status 404 returned error can't find the container with id 030c852650ddeaabb0ec6eb6e6763877ed0e82dabb34f0716e50c144c643db00
Jul 31 18:44:33.303640 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.303612    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh"]
Jul 31 18:44:33.304695 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:33.304670    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0353f16_2658_49f0_b4de_87d3bcf0d529.slice/crio-1d82d5662ece04d486b45734c078be8e1d29ea1040f01a186a57e0fd4ff2aa82 WatchSource:0}: Error finding container 1d82d5662ece04d486b45734c078be8e1d29ea1040f01a186a57e0fd4ff2aa82: Status 404 returned error can't find the container with id 1d82d5662ece04d486b45734c078be8e1d29ea1040f01a186a57e0fd4ff2aa82
Jul 31 18:44:33.396436 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.396408    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5"]
Jul 31 18:44:33.399158 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:33.399127    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7031db7a_a3ee_45f4_8448_4913e8a110c9.slice/crio-affcf56721cc09d0ca1ab4c111cb8a13eb2cc9a3b046a8c033b8c8b9ce608bf3 WatchSource:0}: Error finding container affcf56721cc09d0ca1ab4c111cb8a13eb2cc9a3b046a8c033b8c8b9ce608bf3: Status 404 returned error can't find the container with id affcf56721cc09d0ca1ab4c111cb8a13eb2cc9a3b046a8c033b8c8b9ce608bf3
Jul 31 18:44:33.400964 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.400935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh" event={"ID":"d0353f16-2658-49f0-b4de-87d3bcf0d529","Type":"ContainerStarted","Data":"1d82d5662ece04d486b45734c078be8e1d29ea1040f01a186a57e0fd4ff2aa82"}
Jul 31 18:44:33.402193 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.402167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2" event={"ID":"62dedd37-4d36-4735-b54f-1206da18cb5f","Type":"ContainerStarted","Data":"030c852650ddeaabb0ec6eb6e6763877ed0e82dabb34f0716e50c144c643db00"}
Jul 31 18:44:33.428054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:33.428031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-b2gjc"]
Jul 31 18:44:33.431068 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:33.431031    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da0140a_41df_45f3_bbd3_0cc2d319a729.slice/crio-c27d6895214321cf9541abf12e8f4b62d09788d687587dee2e4d2039f63587bc WatchSource:0}: Error finding container c27d6895214321cf9541abf12e8f4b62d09788d687587dee2e4d2039f63587bc: Status 404 returned error can't find the container with id c27d6895214321cf9541abf12e8f4b62d09788d687587dee2e4d2039f63587bc
Jul 31 18:44:34.408534 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:34.408491    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" event={"ID":"7da0140a-41df-45f3-bbd3-0cc2d319a729","Type":"ContainerStarted","Data":"c27d6895214321cf9541abf12e8f4b62d09788d687587dee2e4d2039f63587bc"}
Jul 31 18:44:34.410052 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:34.410021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5" event={"ID":"7031db7a-a3ee-45f4-8448-4913e8a110c9","Type":"ContainerStarted","Data":"affcf56721cc09d0ca1ab4c111cb8a13eb2cc9a3b046a8c033b8c8b9ce608bf3"}
Jul 31 18:44:38.012073 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.012002    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5rnq2"]
Jul 31 18:44:38.013841 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.013825    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.017454 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.017427    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 18:44:38.017813 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.017793    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 18:44:38.017896 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.017840    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7tf4\""
Jul 31 18:44:38.018867 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.018848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 18:44:38.019073 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.019060    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 18:44:38.035650 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.035627    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rnq2"]
Jul 31 18:44:38.120797 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.120754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca122841-d193-49fa-94e7-e3de5335fe03-data-volume\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.120969 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.120836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.120969 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.120893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.121109 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.120976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxftm\" (UniqueName: \"kubernetes.io/projected/ca122841-d193-49fa-94e7-e3de5335fe03-kube-api-access-rxftm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.121109 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.121020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca122841-d193-49fa-94e7-e3de5335fe03-crio-socket\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222336 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca122841-d193-49fa-94e7-e3de5335fe03-data-volume\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222375    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxftm\" (UniqueName: \"kubernetes.io/projected/ca122841-d193-49fa-94e7-e3de5335fe03-kube-api-access-rxftm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.222497    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.222525    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 18:44:38.222557 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.222558    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:38.72254028 +0000 UTC m=+57.070122141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:38.222982 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.222598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:38.722582988 +0000 UTC m=+57.070164846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : configmap "kube-rbac-proxy" not found
Jul 31 18:44:38.222982 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca122841-d193-49fa-94e7-e3de5335fe03-crio-socket\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222982 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca122841-d193-49fa-94e7-e3de5335fe03-data-volume\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.222982 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.222811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca122841-d193-49fa-94e7-e3de5335fe03-crio-socket\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.232602 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.232584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxftm\" (UniqueName: \"kubernetes.io/projected/ca122841-d193-49fa-94e7-e3de5335fe03-kube-api-access-rxftm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.422723 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.422695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/0.log"
Jul 31 18:44:38.422915 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.422740    2557 generic.go:356] "Generic (PLEG): container finished" podID="7da0140a-41df-45f3-bbd3-0cc2d319a729" containerID="a3cb64642fe2138c56407255979e5f1919d671e578e4f8d531e487e2ba710210" exitCode=255
Jul 31 18:44:38.422915 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.422833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" event={"ID":"7da0140a-41df-45f3-bbd3-0cc2d319a729","Type":"ContainerDied","Data":"a3cb64642fe2138c56407255979e5f1919d671e578e4f8d531e487e2ba710210"}
Jul 31 18:44:38.423103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.423088    2557 scope.go:117] "RemoveContainer" containerID="a3cb64642fe2138c56407255979e5f1919d671e578e4f8d531e487e2ba710210"
Jul 31 18:44:38.424485 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.424400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5" event={"ID":"7031db7a-a3ee-45f4-8448-4913e8a110c9","Type":"ContainerStarted","Data":"9927dc830aa91bdb4aa10c7ef4560d8f651ad1db33009ef4374dceb23f4d429a"}
Jul 31 18:44:38.426126 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.426103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh" event={"ID":"d0353f16-2658-49f0-b4de-87d3bcf0d529","Type":"ContainerStarted","Data":"6915e3f85d1cda07e29f25876bf70d2dbd39e824f9f1d26fb8be94886a043b51"}
Jul 31 18:44:38.427749 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.427728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2" event={"ID":"62dedd37-4d36-4735-b54f-1206da18cb5f","Type":"ContainerStarted","Data":"fdb8ddefbbf009b707820bd16b47ff66e769a4861971dce342fcbef68f1d53c8"}
Jul 31 18:44:38.471071 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.470950    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh" podStartSLOduration=2.115528766 podStartE2EDuration="6.470929716s" totalImagesPullingTime="4.35540095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:38.46983215 +0000 UTC m=+56.817414031" watchObservedRunningTime="2026-07-31 18:44:38.470929716 +0000 UTC m=+56.818511597"
Jul 31 18:44:38.487509 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.487466    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-g5mf5" podStartSLOduration=2.226038879 podStartE2EDuration="6.487431651s" totalImagesPullingTime="4.261392772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:38.486717625 +0000 UTC m=+56.834299521" watchObservedRunningTime="2026-07-31 18:44:38.487431651 +0000 UTC m=+56.835013859"
Jul 31 18:44:38.509044 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.508976    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4l8h2" podStartSLOduration=2.120544563 podStartE2EDuration="6.508952865s" totalImagesPullingTime="4.388408302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:38.508055835 +0000 UTC m=+56.855637718" watchObservedRunningTime="2026-07-31 18:44:38.508952865 +0000 UTC m=+56.856534747"
Jul 31 18:44:38.728642 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.728540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.728821 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.728655    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 18:44:38.728821 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:38.728692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:38.728821 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.728718    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:39.728701801 +0000 UTC m=+58.076283660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : configmap "kube-rbac-proxy" not found
Jul 31 18:44:38.728821 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.728783    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:38.728988 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:38.728838    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:39.728823129 +0000 UTC m=+58.076405003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:39.432707 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.432674    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:44:39.433104 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.433088    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/0.log"
Jul 31 18:44:39.433157 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.433125    2557 generic.go:356] "Generic (PLEG): container finished" podID="7da0140a-41df-45f3-bbd3-0cc2d319a729" containerID="7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2" exitCode=255
Jul 31 18:44:39.433205 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.433185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" event={"ID":"7da0140a-41df-45f3-bbd3-0cc2d319a729","Type":"ContainerDied","Data":"7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2"}
Jul 31 18:44:39.433286 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.433276    2557 scope.go:117] "RemoveContainer" containerID="a3cb64642fe2138c56407255979e5f1919d671e578e4f8d531e487e2ba710210"
Jul 31 18:44:39.433926 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.433906    2557 scope.go:117] "RemoveContainer" containerID="7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2"
Jul 31 18:44:39.434075 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:39.434054    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b2gjc_openshift-console-operator(7da0140a-41df-45f3-bbd3-0cc2d319a729)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" podUID="7da0140a-41df-45f3-bbd3-0cc2d319a729"
Jul 31 18:44:39.736097 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.736014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:39.736097 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.736062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:39.736269 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:39.736148    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 18:44:39.736269 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:39.736164    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:39.736269 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:39.736220    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:41.736201243 +0000 UTC m=+60.083783115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:39.736269 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:39.736237    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:41.736227996 +0000 UTC m=+60.083809854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : configmap "kube-rbac-proxy" not found
Jul 31 18:44:39.756168 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:39.756138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rprxx_f7ed081d-85b1-41da-af82-2caff6b85486/dns-node-resolver/0.log"
Jul 31 18:44:40.349505 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.349476    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d66d8_9f8de46e-3b7b-4896-a7c4-2256edda4630/node-ca/0.log"
Jul 31 18:44:40.436648 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.436610    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:44:40.437007 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.436936    2557 scope.go:117] "RemoveContainer" containerID="7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2"
Jul 31 18:44:40.437098 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:40.437077    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b2gjc_openshift-console-operator(7da0140a-41df-45f3-bbd3-0cc2d319a729)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" podUID="7da0140a-41df-45f3-bbd3-0cc2d319a729"
Jul 31 18:44:40.742921 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.742839    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"]
Jul 31 18:44:40.745369 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.745354    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"
Jul 31 18:44:40.747931 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.747901    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 18:44:40.748040 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.747975    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9lksg\""
Jul 31 18:44:40.748190 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.748173    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 18:44:40.755517 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.755485    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"]
Jul 31 18:44:40.846510 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.846472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdb4\" (UniqueName: \"kubernetes.io/projected/6ddd4be7-e7ba-4f7c-b037-5a7756647d46-kube-api-access-mmdb4\") pod \"migrator-68bc745d87-l6rw2\" (UID: \"6ddd4be7-e7ba-4f7c-b037-5a7756647d46\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"
Jul 31 18:44:40.947918 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.947888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdb4\" (UniqueName: \"kubernetes.io/projected/6ddd4be7-e7ba-4f7c-b037-5a7756647d46-kube-api-access-mmdb4\") pod \"migrator-68bc745d87-l6rw2\" (UID: \"6ddd4be7-e7ba-4f7c-b037-5a7756647d46\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"
Jul 31 18:44:40.957743 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:40.957706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdb4\" (UniqueName: \"kubernetes.io/projected/6ddd4be7-e7ba-4f7c-b037-5a7756647d46-kube-api-access-mmdb4\") pod \"migrator-68bc745d87-l6rw2\" (UID: \"6ddd4be7-e7ba-4f7c-b037-5a7756647d46\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"
Jul 31 18:44:41.054812 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.054741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"
Jul 31 18:44:41.149246 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.149223    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2nvps_8ceefe50-afa0-49e4-ab29-a898d0173807/serve-healthcheck-canary/0.log"
Jul 31 18:44:41.178818 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.178796    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2"]
Jul 31 18:44:41.181218 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:41.181189    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddd4be7_e7ba_4f7c_b037_5a7756647d46.slice/crio-4a83b2fa6d5b8959e12c7b0b3e2a1e5b389c6a333f187624bb467d541667f2f2 WatchSource:0}: Error finding container 4a83b2fa6d5b8959e12c7b0b3e2a1e5b389c6a333f187624bb467d541667f2f2: Status 404 returned error can't find the container with id 4a83b2fa6d5b8959e12c7b0b3e2a1e5b389c6a333f187624bb467d541667f2f2
Jul 31 18:44:41.440226 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.440193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2" event={"ID":"6ddd4be7-e7ba-4f7c-b037-5a7756647d46","Type":"ContainerStarted","Data":"4a83b2fa6d5b8959e12c7b0b3e2a1e5b389c6a333f187624bb467d541667f2f2"}
Jul 31 18:44:41.492391 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.492364    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kd46r"]
Jul 31 18:44:41.494662 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.494648    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.503900 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.503880    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 18:44:41.504180 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.504158    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 18:44:41.504274 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.504194    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 18:44:41.505211 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.505198    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 18:44:41.505271 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.505211    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6mh7h\""
Jul 31 18:44:41.509753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.509728    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kd46r"]
Jul 31 18:44:41.552364 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.552331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7nx\" (UniqueName: \"kubernetes.io/projected/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-kube-api-access-4b7nx\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.552364 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.552365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-key\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.552576 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.552398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-cabundle\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.653869 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.653830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7nx\" (UniqueName: \"kubernetes.io/projected/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-kube-api-access-4b7nx\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.653869 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.653872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-key\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.654119 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.653923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-cabundle\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.654571 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.654547    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-cabundle\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.656510 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.656491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-signing-key\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.664282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.664257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7nx\" (UniqueName: \"kubernetes.io/projected/b457b010-a7b6-4ca9-a62f-508c8aaa8eed-kube-api-access-4b7nx\") pod \"service-ca-98c88fbb5-kd46r\" (UID: \"b457b010-a7b6-4ca9-a62f-508c8aaa8eed\") " pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.755222 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.755141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:41.755222 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.755195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:41.755413 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:41.755309    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:41.755413 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:41.755318    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 18:44:41.755413 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:41.755421    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.755366309 +0000 UTC m=+64.102948187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : configmap "kube-rbac-proxy" not found
Jul 31 18:44:41.755581 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:41.755492    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:45.755432179 +0000 UTC m=+64.103014038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:41.804124 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.804092    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-kd46r"
Jul 31 18:44:41.930104 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:41.930069    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-kd46r"]
Jul 31 18:44:41.933353 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:41.933322    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb457b010_a7b6_4ca9_a62f_508c8aaa8eed.slice/crio-1f3ccb4fc2a2942f7f65a3e5cf4455db3eb498440cf220f1a6c977a1141fc23c WatchSource:0}: Error finding container 1f3ccb4fc2a2942f7f65a3e5cf4455db3eb498440cf220f1a6c977a1141fc23c: Status 404 returned error can't find the container with id 1f3ccb4fc2a2942f7f65a3e5cf4455db3eb498440cf220f1a6c977a1141fc23c
Jul 31 18:44:42.443767 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:42.443740    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-kd46r" event={"ID":"b457b010-a7b6-4ca9-a62f-508c8aaa8eed","Type":"ContainerStarted","Data":"1f3ccb4fc2a2942f7f65a3e5cf4455db3eb498440cf220f1a6c977a1141fc23c"}
Jul 31 18:44:43.237590 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.237544    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:43.237771 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.237666    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:43.238189 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.238172    2557 scope.go:117] "RemoveContainer" containerID="7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2"
Jul 31 18:44:43.238404 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:43.238379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-b2gjc_openshift-console-operator(7da0140a-41df-45f3-bbd3-0cc2d319a729)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" podUID="7da0140a-41df-45f3-bbd3-0cc2d319a729"
Jul 31 18:44:43.449732 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.449687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2" event={"ID":"6ddd4be7-e7ba-4f7c-b037-5a7756647d46","Type":"ContainerStarted","Data":"96dfe1c0c96dab49f85406ee57b632451b638370e2e25da3dfc93fd06e999fa3"}
Jul 31 18:44:43.450143 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.449765    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2" event={"ID":"6ddd4be7-e7ba-4f7c-b037-5a7756647d46","Type":"ContainerStarted","Data":"01ccc4975784878bdf321ea87d179521fc6961d33d0049e86a5ad30b11b1d3e4"}
Jul 31 18:44:43.471665 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:43.471613    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-l6rw2" podStartSLOduration=2.237581149 podStartE2EDuration="3.471596668s" totalImagesPullingTime="1.234015519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:43.47026457 +0000 UTC m=+61.817846470" watchObservedRunningTime="2026-07-31 18:44:43.471596668 +0000 UTC m=+61.819178581"
Jul 31 18:44:44.353744 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:44.353715    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkck9"
Jul 31 18:44:44.454017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:44.453974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-kd46r" event={"ID":"b457b010-a7b6-4ca9-a62f-508c8aaa8eed","Type":"ContainerStarted","Data":"a15c32cfe7c4dfce9e299e174c7d9e9eb149867d51d878cdde55d4d8ff553d3d"}
Jul 31 18:44:44.474530 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:44.474479    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-kd46r" podStartSLOduration=1.314482941 podStartE2EDuration="3.474462212s" totalImagesPullingTime="2.159979271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:44.473756966 +0000 UTC m=+62.821338850" watchObservedRunningTime="2026-07-31 18:44:44.474462212 +0000 UTC m=+62.822044083"
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:45.787504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:45.787587    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:45.787788    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:45.787862    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:53.787834373 +0000 UTC m=+72.135416251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : configmap "kube-rbac-proxy" not found
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:45.788408    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:45.790917 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:45.788490    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:44:53.78847356 +0000 UTC m=+72.136055432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:47.909199 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:47.909167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:44:47.911950 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:47.911925    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 18:44:47.920038 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:47.920013    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 18:44:47.920141 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:47.920080    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs podName:53ebbbe8-1bcd-49e4-9f36-f95c74d6534b nodeName:}" failed. No retries permitted until 2026-07-31 18:45:51.920062283 +0000 UTC m=+130.267644154 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs") pod "network-metrics-daemon-stg44" (UID: "53ebbbe8-1bcd-49e4-9f36-f95c74d6534b") : secret "metrics-daemon-secret" not found
Jul 31 18:44:48.009568 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.009531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:48.012197 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.012179    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 18:44:48.022693 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.022670    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df4356d7-969b-4b93-a3e6-3fb45690cd29-original-pull-secret\") pod \"global-pull-secret-syncer-fhf8w\" (UID: \"df4356d7-969b-4b93-a3e6-3fb45690cd29\") " pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:48.110909 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.110862    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:48.113563 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.113535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdlj\" (UniqueName: \"kubernetes.io/projected/2ce52429-d43f-4e1c-b2d1-2825b30277b7-kube-api-access-4pdlj\") pod \"network-check-target-5gnwc\" (UID: \"2ce52429-d43f-4e1c-b2d1-2825b30277b7\") " pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:48.291471 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.291353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhf8w"
Jul 31 18:44:48.308686 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.308663    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-99tgj\""
Jul 31 18:44:48.316395 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.316366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:48.432433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.432398    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhf8w"]
Jul 31 18:44:48.435374 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:48.435340    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4356d7_969b_4b93_a3e6_3fb45690cd29.slice/crio-41286576f18e2447106f86acbf9e990d7c7e36a27e084daef29124a6c64857db WatchSource:0}: Error finding container 41286576f18e2447106f86acbf9e990d7c7e36a27e084daef29124a6c64857db: Status 404 returned error can't find the container with id 41286576f18e2447106f86acbf9e990d7c7e36a27e084daef29124a6c64857db
Jul 31 18:44:48.453918 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.453862    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gnwc"]
Jul 31 18:44:48.456128 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:44:48.456103    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce52429_d43f_4e1c_b2d1_2825b30277b7.slice/crio-afdf64ab8bea25f93c2e6e3d416b157d874a9574229ed9c4953f023bb655d09a WatchSource:0}: Error finding container afdf64ab8bea25f93c2e6e3d416b157d874a9574229ed9c4953f023bb655d09a: Status 404 returned error can't find the container with id afdf64ab8bea25f93c2e6e3d416b157d874a9574229ed9c4953f023bb655d09a
Jul 31 18:44:48.468504 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.468473    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gnwc" event={"ID":"2ce52429-d43f-4e1c-b2d1-2825b30277b7","Type":"ContainerStarted","Data":"afdf64ab8bea25f93c2e6e3d416b157d874a9574229ed9c4953f023bb655d09a"}
Jul 31 18:44:48.469604 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:48.469586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhf8w" event={"ID":"df4356d7-969b-4b93-a3e6-3fb45690cd29","Type":"ContainerStarted","Data":"41286576f18e2447106f86acbf9e990d7c7e36a27e084daef29124a6c64857db"}
Jul 31 18:44:49.221214 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:49.221173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:44:49.221214 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:49.221226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:44:49.221731 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:49.221331    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 18:44:49.221731 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:49.221352    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 18:44:49.221731 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:49.221374    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b56fcd8c-7z4ng: secret "image-registry-tls" not found
Jul 31 18:44:49.221731 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:49.221390    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls podName:5a0497e7-04d8-4120-81fe-23357e4d9b1b nodeName:}" failed. No retries permitted until 2026-07-31 18:45:21.221372416 +0000 UTC m=+99.568954296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls") pod "dns-default-5ps6l" (UID: "5a0497e7-04d8-4120-81fe-23357e4d9b1b") : secret "dns-default-metrics-tls" not found
Jul 31 18:44:49.221731 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:49.221431    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls podName:a946c65f-8213-4b86-85f3-c2605728236f nodeName:}" failed. No retries permitted until 2026-07-31 18:45:21.221409958 +0000 UTC m=+99.568991820 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls") pod "image-registry-78b56fcd8c-7z4ng" (UID: "a946c65f-8213-4b86-85f3-c2605728236f") : secret "image-registry-tls" not found
Jul 31 18:44:49.475261 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:49.475168    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gnwc" event={"ID":"2ce52429-d43f-4e1c-b2d1-2825b30277b7","Type":"ContainerStarted","Data":"b4248597c7425dded384d534768510e24d7e228aac2c66658adc194ced992fef"}
Jul 31 18:44:49.475420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:49.475320    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:44:49.496947 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:49.496877    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gnwc" podStartSLOduration=66.49685809100001 podStartE2EDuration="1m6.496858091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:49.496321747 +0000 UTC m=+67.843903628" watchObservedRunningTime="2026-07-31 18:44:49.496858091 +0000 UTC m=+67.844439963"
Jul 31 18:44:53.490121 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:53.490032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhf8w" event={"ID":"df4356d7-969b-4b93-a3e6-3fb45690cd29","Type":"ContainerStarted","Data":"379bcf7669603c1f0e718a2fe4605c86278b8ac533402497988a17884b13e584"}
Jul 31 18:44:53.509777 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:53.509726    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fhf8w" podStartSLOduration=65.836423453 podStartE2EDuration="1m10.509708968s" totalImagesPullingTime="4.673285515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:53.508609817 +0000 UTC m=+71.856191698" watchObservedRunningTime="2026-07-31 18:44:53.509708968 +0000 UTC m=+71.857290847"
Jul 31 18:44:53.865194 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:53.865155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:53.865363 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:53.865206    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:53.865363 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:53.865311    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:53.865483 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:44:53.865377    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls podName:ca122841-d193-49fa-94e7-e3de5335fe03 nodeName:}" failed. No retries permitted until 2026-07-31 18:45:09.865361491 +0000 UTC m=+88.212943371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rnq2" (UID: "ca122841-d193-49fa-94e7-e3de5335fe03") : secret "insights-runtime-extractor-tls" not found
Jul 31 18:44:53.865754 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:53.865737    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca122841-d193-49fa-94e7-e3de5335fe03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:44:57.178685 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.178646    2557 scope.go:117] "RemoveContainer" containerID="7f4d8ab33b5471a0d2e3b670f053717bf9239aa7f4052c0e485ffaf9cf5cf8b2"
Jul 31 18:44:57.505597 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.505515    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:44:57.505751 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.505639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" event={"ID":"7da0140a-41df-45f3-bbd3-0cc2d319a729","Type":"ContainerStarted","Data":"fb3812b8ef947af71535ea08982319659ec569a0c2f85cb638a4dcd35f62c61b"}
Jul 31 18:44:57.505990 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.505956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:44:57.528156 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.528109    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc" podStartSLOduration=21.292793212 podStartE2EDuration="25.52809492s" totalImagesPullingTime="4.235301708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:44:57.527140496 +0000 UTC m=+75.874722378" watchObservedRunningTime="2026-07-31 18:44:57.52809492 +0000 UTC m=+75.875676831"
Jul 31 18:44:57.688934 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:44:57.688905    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-b2gjc"
Jul 31 18:45:01.988700 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.988666    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"]
Jul 31 18:45:01.991775 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.991754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:01.994918 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.994891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 18:45:01.995936 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.995902    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 18:45:01.996315 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.996293    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 18:45:02.000587 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.996432    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 18:45:02.000587 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.996305    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 18:45:02.000587 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:01.996711    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 18:45:02.001904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.001884    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 18:45:02.014309 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.014266    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"]
Jul 31 18:45:02.139239 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.139427 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfrc\" (UniqueName: \"kubernetes.io/projected/b810bdf3-f09d-47d7-9665-7f2f6db713cd-kube-api-access-fvfrc\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.139427 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.139577 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b810bdf3-f09d-47d7-9665-7f2f6db713cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.139577 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.139683 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.139574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240186 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfrc\" (UniqueName: \"kubernetes.io/projected/b810bdf3-f09d-47d7-9665-7f2f6db713cd-kube-api-access-fvfrc\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240186 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240371 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b810bdf3-f09d-47d7-9665-7f2f6db713cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240407 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240478 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240410    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.240478 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.240470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.241267 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.241239    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b810bdf3-f09d-47d7-9665-7f2f6db713cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.242937 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.242879    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.243033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.242968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.243077 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.243030    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-ca\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.243229 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.243201    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b810bdf3-f09d-47d7-9665-7f2f6db713cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.249763 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.249743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfrc\" (UniqueName: \"kubernetes.io/projected/b810bdf3-f09d-47d7-9665-7f2f6db713cd-kube-api-access-fvfrc\") pod \"cluster-proxy-proxy-agent-7bd786dbd4-7hxjc\" (UID: \"b810bdf3-f09d-47d7-9665-7f2f6db713cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.316931 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.316896    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"
Jul 31 18:45:02.447282 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.447253    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc"]
Jul 31 18:45:02.450031 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:02.450007    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb810bdf3_f09d_47d7_9665_7f2f6db713cd.slice/crio-9d86b1e0bdc41ea1bf2640de30ee86f541a99f41b9da9e7b4dcf7042c90317d5 WatchSource:0}: Error finding container 9d86b1e0bdc41ea1bf2640de30ee86f541a99f41b9da9e7b4dcf7042c90317d5: Status 404 returned error can't find the container with id 9d86b1e0bdc41ea1bf2640de30ee86f541a99f41b9da9e7b4dcf7042c90317d5
Jul 31 18:45:02.522184 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:02.522087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc" event={"ID":"b810bdf3-f09d-47d7-9665-7f2f6db713cd","Type":"ContainerStarted","Data":"9d86b1e0bdc41ea1bf2640de30ee86f541a99f41b9da9e7b4dcf7042c90317d5"}
Jul 31 18:45:08.544380 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:08.544342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc" event={"ID":"b810bdf3-f09d-47d7-9665-7f2f6db713cd","Type":"ContainerStarted","Data":"6c63e07a3b66e2583575f5720b64e5c80a1b77e2ef54e33ab9d991eb504e77dc"}
Jul 31 18:45:08.544380 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:08.544387    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc" event={"ID":"b810bdf3-f09d-47d7-9665-7f2f6db713cd","Type":"ContainerStarted","Data":"b2eb03afc5eb86ef938c9b693f44f9ffcba9eddf33492e08c5e563860744d5fc"}
Jul 31 18:45:08.544807 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:08.544400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc" event={"ID":"b810bdf3-f09d-47d7-9665-7f2f6db713cd","Type":"ContainerStarted","Data":"e4613a356ca7943e6d9e0b7af1fbb2b54afddef8f89364d4ffd82adeeddf35aa"}
Jul 31 18:45:08.570498 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:08.570423    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7bd786dbd4-7hxjc" podStartSLOduration=2.51854094 podStartE2EDuration="7.570409043s" totalImagesPullingTime="5.051868103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:08.568413256 +0000 UTC m=+86.915995135" watchObservedRunningTime="2026-07-31 18:45:08.570409043 +0000 UTC m=+86.917990923"
Jul 31 18:45:09.906187 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:09.906150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:45:09.908834 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:09.908802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca122841-d193-49fa-94e7-e3de5335fe03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rnq2\" (UID: \"ca122841-d193-49fa-94e7-e3de5335fe03\") " pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:45:10.128777 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:10.128743    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7tf4\""
Jul 31 18:45:10.135167 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:10.135137    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rnq2"
Jul 31 18:45:10.306751 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:10.306574    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rnq2"]
Jul 31 18:45:10.309385 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:10.309360    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca122841_d193_49fa_94e7_e3de5335fe03.slice/crio-c8be1aa4a209aa62501670d83d79364ad49453db443683cdc4d1d92ff0dcf72c WatchSource:0}: Error finding container c8be1aa4a209aa62501670d83d79364ad49453db443683cdc4d1d92ff0dcf72c: Status 404 returned error can't find the container with id c8be1aa4a209aa62501670d83d79364ad49453db443683cdc4d1d92ff0dcf72c
Jul 31 18:45:10.552199 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:10.552162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rnq2" event={"ID":"ca122841-d193-49fa-94e7-e3de5335fe03","Type":"ContainerStarted","Data":"e944785070ae1560dcd7b7e2b25a5a4d9fc672d8c978ea532f3acde44027ef49"}
Jul 31 18:45:10.552363 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:10.552217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rnq2" event={"ID":"ca122841-d193-49fa-94e7-e3de5335fe03","Type":"ContainerStarted","Data":"c8be1aa4a209aa62501670d83d79364ad49453db443683cdc4d1d92ff0dcf72c"}
Jul 31 18:45:11.560172 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:11.560138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rnq2" event={"ID":"ca122841-d193-49fa-94e7-e3de5335fe03","Type":"ContainerStarted","Data":"16f57ff551574864e360c611118bafb57715216ddf22df6940a34370683a33d6"}
Jul 31 18:45:13.572547 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:13.572513    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rnq2" event={"ID":"ca122841-d193-49fa-94e7-e3de5335fe03","Type":"ContainerStarted","Data":"e536f920de9ff76f56a08b3d3ca52941bf759d06338694546d91144d7288c994"}
Jul 31 18:45:13.595413 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:13.595366    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5rnq2" podStartSLOduration=34.290371591 podStartE2EDuration="36.595350201s" totalImagesPullingTime="2.30497861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 18:44:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:13.59425151 +0000 UTC m=+91.941833388" watchObservedRunningTime="2026-07-31 18:45:13.595350201 +0000 UTC m=+91.942932072"
Jul 31 18:45:18.195762 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.195729    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-29kkd"]
Jul 31 18:45:18.201798 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.201776    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.205901 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:45:18.205866    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-29kkd\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object" podUID="706d6d31-2fab-47d2-a1f2-36e95622be06" pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.206038 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:45:18.205923    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 31 18:45:18.211491 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:45:18.211017    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Jul 31 18:45:18.211633 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:45:18.211559    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Jul 31 18:45:18.211958 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.211939    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 18:45:18.212546 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.212529    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbzjz\""
Jul 31 18:45:18.212634 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.212563    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 18:45:18.212856 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.212841    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 18:45:18.376184 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-sys\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376184 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc7s\" (UniqueName: \"kubernetes.io/projected/706d6d31-2fab-47d2-a1f2-36e95622be06-kube-api-access-9mc7s\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-root\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-metrics-client-ca\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-textfile\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-accelerators-collector-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376433 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376402    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-tls\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.376759 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.376480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-wtmp\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477094 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.476996    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-root\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477094 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-metrics-client-ca\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477094 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-textfile\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477120    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-root\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-accelerators-collector-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-tls\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-wtmp\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477602 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-sys\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477602 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc7s\" (UniqueName: \"kubernetes.io/projected/706d6d31-2fab-47d2-a1f2-36e95622be06-kube-api-access-9mc7s\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477717 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477690    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-wtmp\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477814 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-metrics-client-ca\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.477874 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477823    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/706d6d31-2fab-47d2-a1f2-36e95622be06-sys\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.478010 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.477991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-textfile\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:18.480073 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:18.480048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:19.108875 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.108848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 18:45:19.118981 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.118950    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-accelerators-collector-config\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:19.311958 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.311932    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 18:45:19.321182 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.321146    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/706d6d31-2fab-47d2-a1f2-36e95622be06-node-exporter-tls\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:19.375671 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.375607    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 18:45:19.387803 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.387781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc7s\" (UniqueName: \"kubernetes.io/projected/706d6d31-2fab-47d2-a1f2-36e95622be06-kube-api-access-9mc7s\") pod \"node-exporter-29kkd\" (UID: \"706d6d31-2fab-47d2-a1f2-36e95622be06\") " pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:19.413948 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.413921    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29kkd"
Jul 31 18:45:19.426783 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:19.426752    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706d6d31_2fab_47d2_a1f2_36e95622be06.slice/crio-1a64724e4ab6da4ebf3515064988c678f5a0c6359118163253f6e4c651dc1e4d WatchSource:0}: Error finding container 1a64724e4ab6da4ebf3515064988c678f5a0c6359118163253f6e4c651dc1e4d: Status 404 returned error can't find the container with id 1a64724e4ab6da4ebf3515064988c678f5a0c6359118163253f6e4c651dc1e4d
Jul 31 18:45:19.593340 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:19.593308    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29kkd" event={"ID":"706d6d31-2fab-47d2-a1f2-36e95622be06","Type":"ContainerStarted","Data":"1a64724e4ab6da4ebf3515064988c678f5a0c6359118163253f6e4c651dc1e4d"}
Jul 31 18:45:20.481674 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:20.481646    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gnwc"
Jul 31 18:45:20.597969 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:20.597931    2557 generic.go:356] "Generic (PLEG): container finished" podID="706d6d31-2fab-47d2-a1f2-36e95622be06" containerID="fe907510b2ef6fe6b88f26f022d883d45330aec3a1f48fe73012812c50629f3d" exitCode=0
Jul 31 18:45:20.598114 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:20.597989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29kkd" event={"ID":"706d6d31-2fab-47d2-a1f2-36e95622be06","Type":"ContainerDied","Data":"fe907510b2ef6fe6b88f26f022d883d45330aec3a1f48fe73012812c50629f3d"}
Jul 31 18:45:21.306086 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.306044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:21.306086 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.306087    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:45:21.308734 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.308710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"image-registry-78b56fcd8c-7z4ng\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") " pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:21.308831 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.308712    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a0497e7-04d8-4120-81fe-23357e4d9b1b-metrics-tls\") pod \"dns-default-5ps6l\" (UID: \"5a0497e7-04d8-4120-81fe-23357e4d9b1b\") " pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:45:21.573862 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.573786    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4pbfx\""
Jul 31 18:45:21.581926 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.581908    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:21.590574 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.590544    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8wjd\""
Jul 31 18:45:21.598138 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.598116    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:45:21.606593 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.606565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29kkd" event={"ID":"706d6d31-2fab-47d2-a1f2-36e95622be06","Type":"ContainerStarted","Data":"a664eae63fce87930bcbf4dc98c8656eb36c0b673384399537be09a4c7f8d826"}
Jul 31 18:45:21.606690 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.606649    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29kkd" event={"ID":"706d6d31-2fab-47d2-a1f2-36e95622be06","Type":"ContainerStarted","Data":"442e4ff310534e9c5c992d5e75c54e3346d1086f2df482a6f2f5eb3be7f09d97"}
Jul 31 18:45:21.627384 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.627336    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-29kkd" podStartSLOduration=2.601616776 podStartE2EDuration="3.627321079s" totalImagesPullingTime="954.0207ms" totalInitContainerRuntime="71.683603ms" isStatefulPod=true podCreationTimestamp="2026-07-31 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:21.626613685 +0000 UTC m=+99.974195566" watchObservedRunningTime="2026-07-31 18:45:21.627321079 +0000 UTC m=+99.974902960"
Jul 31 18:45:21.743073 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.743040    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:45:21.746021 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:21.745997    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda946c65f_8213_4b86_85f3_c2605728236f.slice/crio-2e8f63f6e5702ccb885b37c422e8bb959186ade485342886c97f3a857b7b6fa7 WatchSource:0}: Error finding container 2e8f63f6e5702ccb885b37c422e8bb959186ade485342886c97f3a857b7b6fa7: Status 404 returned error can't find the container with id 2e8f63f6e5702ccb885b37c422e8bb959186ade485342886c97f3a857b7b6fa7
Jul 31 18:45:21.758784 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:21.758761    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ps6l"]
Jul 31 18:45:21.761588 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:21.761562    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0497e7_04d8_4120_81fe_23357e4d9b1b.slice/crio-33395f92b5011696c7d93b355e85fd03965454f7d4da85630c6192e2d8f113fa WatchSource:0}: Error finding container 33395f92b5011696c7d93b355e85fd03965454f7d4da85630c6192e2d8f113fa: Status 404 returned error can't find the container with id 33395f92b5011696c7d93b355e85fd03965454f7d4da85630c6192e2d8f113fa
Jul 31 18:45:22.613340 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:22.613274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" event={"ID":"a946c65f-8213-4b86-85f3-c2605728236f","Type":"ContainerStarted","Data":"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"}
Jul 31 18:45:22.613340 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:22.613339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" event={"ID":"a946c65f-8213-4b86-85f3-c2605728236f","Type":"ContainerStarted","Data":"2e8f63f6e5702ccb885b37c422e8bb959186ade485342886c97f3a857b7b6fa7"}
Jul 31 18:45:22.613876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:22.613502    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:22.615514 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:22.614918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ps6l" event={"ID":"5a0497e7-04d8-4120-81fe-23357e4d9b1b","Type":"ContainerStarted","Data":"33395f92b5011696c7d93b355e85fd03965454f7d4da85630c6192e2d8f113fa"}
Jul 31 18:45:22.655340 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:22.655292    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" podStartSLOduration=99.655263551 podStartE2EDuration="1m39.655263551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:22.654032208 +0000 UTC m=+101.001614088" watchObservedRunningTime="2026-07-31 18:45:22.655263551 +0000 UTC m=+101.002845448"
Jul 31 18:45:24.412032 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:24.411998    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:45:24.624898 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:24.624861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ps6l" event={"ID":"5a0497e7-04d8-4120-81fe-23357e4d9b1b","Type":"ContainerStarted","Data":"b7978df3f811433152111319990c865283a9a80360df7402892b8b0de216312d"}
Jul 31 18:45:24.625090 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:24.624925    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ps6l" event={"ID":"5a0497e7-04d8-4120-81fe-23357e4d9b1b","Type":"ContainerStarted","Data":"20a661b8ccbc221fe880a737c2d8ea0261d46e728e87fac26f21e627deed34b8"}
Jul 31 18:45:24.625199 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:24.625182    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:45:24.695957 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:24.695858    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5ps6l" podStartSLOduration=65.501327185 podStartE2EDuration="1m7.695842779s" totalImagesPullingTime="2.194515594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:24.694361101 +0000 UTC m=+103.041943004" watchObservedRunningTime="2026-07-31 18:45:24.695842779 +0000 UTC m=+103.043424659"
Jul 31 18:45:34.630065 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:34.630037    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5ps6l"
Jul 31 18:45:44.631255 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:44.631228    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:44.696163 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:44.696132    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0353f16-2658-49f0-b4de-87d3bcf0d529" containerID="6915e3f85d1cda07e29f25876bf70d2dbd39e824f9f1d26fb8be94886a043b51" exitCode=0
Jul 31 18:45:44.696326 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:44.696206    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh" event={"ID":"d0353f16-2658-49f0-b4de-87d3bcf0d529","Type":"ContainerDied","Data":"6915e3f85d1cda07e29f25876bf70d2dbd39e824f9f1d26fb8be94886a043b51"}
Jul 31 18:45:44.696544 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:44.696530    2557 scope.go:117] "RemoveContainer" containerID="6915e3f85d1cda07e29f25876bf70d2dbd39e824f9f1d26fb8be94886a043b51"
Jul 31 18:45:45.701160 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:45.701124    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-lqgxh" event={"ID":"d0353f16-2658-49f0-b4de-87d3bcf0d529","Type":"ContainerStarted","Data":"dc0da93317a6c4698ad07085d42db28b3e1a8a58d51cc8b4b84edb15276d3470"}
Jul 31 18:45:49.643934 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.643883    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" podUID="a946c65f-8213-4b86-85f3-c2605728236f" containerName="registry" containerID="cri-o://894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8" gracePeriod=30
Jul 31 18:45:49.888011 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.887983    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:49.924784 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.924711    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.924784 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.924771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dh54\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925003 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.924799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925003 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.924927    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925116 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925022    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925116 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925055    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925116 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925081    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925116 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925103    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") pod \"a946c65f-8213-4b86-85f3-c2605728236f\" (UID: \"a946c65f-8213-4b86-85f3-c2605728236f\") "
Jul 31 18:45:49.925675 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925624    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 18:45:49.925807 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.925786    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 18:45:49.928198 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.928146    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 18:45:49.928303 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.928237    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:45:49.928303 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.928277    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 18:45:49.928610 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.928585    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54" (OuterVolumeSpecName: "kube-api-access-4dh54") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "kube-api-access-4dh54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:45:49.928883 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.928868    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:45:49.934354 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:49.934310    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a946c65f-8213-4b86-85f3-c2605728236f" (UID: "a946c65f-8213-4b86-85f3-c2605728236f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:45:50.026638 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026596    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a946c65f-8213-4b86-85f3-c2605728236f-ca-trust-extracted\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026638 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026630    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-image-registry-private-configuration\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026638 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026642    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-bound-sa-token\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026638 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026651    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026660    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a946c65f-8213-4b86-85f3-c2605728236f-installation-pull-secrets\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026671    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dh54\" (UniqueName: \"kubernetes.io/projected/a946c65f-8213-4b86-85f3-c2605728236f-kube-api-access-4dh54\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026679    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-registry-certificates\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.026904 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.026688    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a946c65f-8213-4b86-85f3-c2605728236f-trusted-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:45:50.721130 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.721097    2557 generic.go:356] "Generic (PLEG): container finished" podID="a946c65f-8213-4b86-85f3-c2605728236f" containerID="894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8" exitCode=0
Jul 31 18:45:50.721571 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.721149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" event={"ID":"a946c65f-8213-4b86-85f3-c2605728236f","Type":"ContainerDied","Data":"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"}
Jul 31 18:45:50.721571 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.721159    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng"
Jul 31 18:45:50.721571 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.721216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b56fcd8c-7z4ng" event={"ID":"a946c65f-8213-4b86-85f3-c2605728236f","Type":"ContainerDied","Data":"2e8f63f6e5702ccb885b37c422e8bb959186ade485342886c97f3a857b7b6fa7"}
Jul 31 18:45:50.721571 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.721237    2557 scope.go:117] "RemoveContainer" containerID="894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"
Jul 31 18:45:50.729178 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.729160    2557 scope.go:117] "RemoveContainer" containerID="894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"
Jul 31 18:45:50.729506 ip-10-0-128-163 kubenswrapper[2557]: E0731 18:45:50.729485    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8\": container with ID starting with 894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8 not found: ID does not exist" containerID="894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"
Jul 31 18:45:50.729558 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.729515    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8"} err="failed to get container status \"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8\": rpc error: code = NotFound desc = could not find container \"894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8\": container with ID starting with 894caac3a81c11989ca2e2fef6c2d041d6f637b67136302ae13aaefc0b99dcc8 not found: ID does not exist"
Jul 31 18:45:50.748738 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.748717    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:45:50.754271 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:50.754254    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78b56fcd8c-7z4ng"]
Jul 31 18:45:51.943011 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:51.942974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:45:51.945350 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:51.945328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ebbbe8-1bcd-49e4-9f36-f95c74d6534b-metrics-certs\") pod \"network-metrics-daemon-stg44\" (UID: \"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b\") " pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:45:52.182313 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:52.182284    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a946c65f-8213-4b86-85f3-c2605728236f" path="/var/lib/kubelet/pods/a946c65f-8213-4b86-85f3-c2605728236f/volumes"
Jul 31 18:45:52.201581 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:52.201527    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4mbt\""
Jul 31 18:45:52.209673 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:52.209655    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-stg44"
Jul 31 18:45:52.338913 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:52.338881    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-stg44"]
Jul 31 18:45:52.342122 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:45:52.342095    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ebbbe8_1bcd_49e4_9f36_f95c74d6534b.slice/crio-deb1459c7c2d5d5c13ae1caaefa9bafaa1c9f6e1afb60aaf543636d17827c3f5 WatchSource:0}: Error finding container deb1459c7c2d5d5c13ae1caaefa9bafaa1c9f6e1afb60aaf543636d17827c3f5: Status 404 returned error can't find the container with id deb1459c7c2d5d5c13ae1caaefa9bafaa1c9f6e1afb60aaf543636d17827c3f5
Jul 31 18:45:52.730270 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:52.730236    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stg44" event={"ID":"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b","Type":"ContainerStarted","Data":"deb1459c7c2d5d5c13ae1caaefa9bafaa1c9f6e1afb60aaf543636d17827c3f5"}
Jul 31 18:45:54.739091 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:54.739064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stg44" event={"ID":"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b","Type":"ContainerStarted","Data":"33dfefc22df803500242472991a802913c13be7aec0cfe7b192a6c6e18a218e1"}
Jul 31 18:45:54.739435 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:54.739148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-stg44" event={"ID":"53ebbbe8-1bcd-49e4-9f36-f95c74d6534b","Type":"ContainerStarted","Data":"2851ff0b888915cb11befa8e1354a856cc638e65f6cc80558e26e612c995fb40"}
Jul 31 18:45:54.757575 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:45:54.757533    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-stg44" podStartSLOduration=129.541702833 podStartE2EDuration="2m11.757517539s" totalImagesPullingTime="2.215814706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:45:54.756095439 +0000 UTC m=+133.103677319" watchObservedRunningTime="2026-07-31 18:45:54.757517539 +0000 UTC m=+133.105099453"
Jul 31 18:47:51.635589 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.635556    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"]
Jul 31 18:47:51.636031 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.635803    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a946c65f-8213-4b86-85f3-c2605728236f" containerName="registry"
Jul 31 18:47:51.636031 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.635815    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a946c65f-8213-4b86-85f3-c2605728236f" containerName="registry"
Jul 31 18:47:51.636031 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.635855    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a946c65f-8213-4b86-85f3-c2605728236f" containerName="registry"
Jul 31 18:47:51.638826 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.638807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.641739 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.641715    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 18:47:51.641862 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.641757    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 18:47:51.642810 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.642794    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cz2bg\""
Jul 31 18:47:51.648821 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.648799    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"]
Jul 31 18:47:51.698248 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.698213    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82gh\" (UniqueName: \"kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.698406 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.698255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.698406 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.698335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.799333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.799297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.799333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.799338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82gh\" (UniqueName: \"kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.799556 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.799379    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.799699 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.799681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.799772 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.799754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.810035 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.810009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82gh\" (UniqueName: \"kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:51.948228 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:51.948127    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:47:52.078989 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:52.078943    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"]
Jul 31 18:47:52.081507 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:47:52.081482    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35930d2b_26fc_4d1a_a1d1_760ddada5212.slice/crio-25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c WatchSource:0}: Error finding container 25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c: Status 404 returned error can't find the container with id 25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c
Jul 31 18:47:52.151721 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:52.151686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk" event={"ID":"35930d2b-26fc-4d1a-a1d1-760ddada5212","Type":"ContainerStarted","Data":"25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c"}
Jul 31 18:47:59.177831 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:59.177802    2557 generic.go:356] "Generic (PLEG): container finished" podID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerID="581d395d0f74fc2c6d720e30de6d19e4797e077c219d928131a7878892333d53" exitCode=0
Jul 31 18:47:59.178251 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:47:59.177893    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk" event={"ID":"35930d2b-26fc-4d1a-a1d1-760ddada5212","Type":"ContainerDied","Data":"581d395d0f74fc2c6d720e30de6d19e4797e077c219d928131a7878892333d53"}
Jul 31 18:48:02.189241 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:02.189209    2557 generic.go:356] "Generic (PLEG): container finished" podID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerID="b387ed90d6b3df191656e977aaddaa09d6fa84d7d515a2432facb9966d22a15a" exitCode=0
Jul 31 18:48:02.189635 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:02.189291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk" event={"ID":"35930d2b-26fc-4d1a-a1d1-760ddada5212","Type":"ContainerDied","Data":"b387ed90d6b3df191656e977aaddaa09d6fa84d7d515a2432facb9966d22a15a"}
Jul 31 18:48:09.223679 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:09.223648    2557 generic.go:356] "Generic (PLEG): container finished" podID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerID="24cd78210d5792a28a1e50c6c5b6daf5410d90ff36d09f42e97138d7b37eb6af" exitCode=0
Jul 31 18:48:09.224018 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:09.223722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk" event={"ID":"35930d2b-26fc-4d1a-a1d1-760ddada5212","Type":"ContainerDied","Data":"24cd78210d5792a28a1e50c6c5b6daf5410d90ff36d09f42e97138d7b37eb6af"}
Jul 31 18:48:10.354600 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.354575    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:48:10.446030 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.445994    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82gh\" (UniqueName: \"kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh\") pod \"35930d2b-26fc-4d1a-a1d1-760ddada5212\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") "
Jul 31 18:48:10.446185 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.446059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle\") pod \"35930d2b-26fc-4d1a-a1d1-760ddada5212\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") "
Jul 31 18:48:10.446185 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.446082    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util\") pod \"35930d2b-26fc-4d1a-a1d1-760ddada5212\" (UID: \"35930d2b-26fc-4d1a-a1d1-760ddada5212\") "
Jul 31 18:48:10.446716 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.446693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle" (OuterVolumeSpecName: "bundle") pod "35930d2b-26fc-4d1a-a1d1-760ddada5212" (UID: "35930d2b-26fc-4d1a-a1d1-760ddada5212"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:10.448342 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.448320    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh" (OuterVolumeSpecName: "kube-api-access-h82gh") pod "35930d2b-26fc-4d1a-a1d1-760ddada5212" (UID: "35930d2b-26fc-4d1a-a1d1-760ddada5212"). InnerVolumeSpecName "kube-api-access-h82gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:48:10.450382 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.450358    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util" (OuterVolumeSpecName: "util") pod "35930d2b-26fc-4d1a-a1d1-760ddada5212" (UID: "35930d2b-26fc-4d1a-a1d1-760ddada5212"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:10.547005 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.546911    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:10.547005 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.546944    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35930d2b-26fc-4d1a-a1d1-760ddada5212-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:10.547005 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:10.546953    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82gh\" (UniqueName: \"kubernetes.io/projected/35930d2b-26fc-4d1a-a1d1-760ddada5212-kube-api-access-h82gh\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:11.233070 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:11.233037    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk" event={"ID":"35930d2b-26fc-4d1a-a1d1-760ddada5212","Type":"ContainerDied","Data":"25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c"}
Jul 31 18:48:11.233070 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:11.233070    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406s7hk"
Jul 31 18:48:11.233263 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:11.233082    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25cdf28d46671783b0ed5f7480e807ef9f72b7d41d1071d9985d3fe4f3a9472c"
Jul 31 18:48:18.828458 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828411    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"]
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828680    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="util"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828691    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="util"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828700    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="extract"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828706    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="extract"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828718    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="pull"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828723    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="pull"
Jul 31 18:48:18.828876 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.828765    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="35930d2b-26fc-4d1a-a1d1-760ddada5212" containerName="extract"
Jul 31 18:48:18.832707 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.832686    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:18.838555 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.838532    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 18:48:18.838679 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.838610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cz2bg\""
Jul 31 18:48:18.839707 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.839689    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 18:48:18.845745 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:18.845726    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"]
Jul 31 18:48:19.008984 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.008928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.009180 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.009004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.009180 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.009062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrrm\" (UniqueName: \"kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.110482 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.110418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.110674 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.110493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrrm\" (UniqueName: \"kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.110674 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.110579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.110976 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.110956    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.111048 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.110953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.120934 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.120911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrrm\" (UniqueName: \"kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.141790 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.141764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:19.272066 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:19.272037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"]
Jul 31 18:48:19.275479 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:48:19.275433    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db23ebd_f6db_4f11_b547_28ea2a57ec36.slice/crio-a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d WatchSource:0}: Error finding container a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d: Status 404 returned error can't find the container with id a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d
Jul 31 18:48:20.268317 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:20.268284    2557 generic.go:356] "Generic (PLEG): container finished" podID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerID="fa298e146ba9c6040842677b78df0bde030d249c596d2a9a4847d732d4b0823e" exitCode=0
Jul 31 18:48:20.268317 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:20.268320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl" event={"ID":"8db23ebd-f6db-4f11-b547-28ea2a57ec36","Type":"ContainerDied","Data":"fa298e146ba9c6040842677b78df0bde030d249c596d2a9a4847d732d4b0823e"}
Jul 31 18:48:20.268748 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:20.268352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl" event={"ID":"8db23ebd-f6db-4f11-b547-28ea2a57ec36","Type":"ContainerStarted","Data":"a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d"}
Jul 31 18:48:23.281296 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:23.281215    2557 generic.go:356] "Generic (PLEG): container finished" podID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerID="db5156011eab82af2c2a65287e7f3551cbc058285887055fe41bdec8c1bbcfd8" exitCode=0
Jul 31 18:48:23.281656 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:23.281306    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl" event={"ID":"8db23ebd-f6db-4f11-b547-28ea2a57ec36","Type":"ContainerDied","Data":"db5156011eab82af2c2a65287e7f3551cbc058285887055fe41bdec8c1bbcfd8"}
Jul 31 18:48:24.287347 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:24.287309    2557 generic.go:356] "Generic (PLEG): container finished" podID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerID="c6036177821707208f25a70281b873fd60fb0ea213378d07cd87375838228338" exitCode=0
Jul 31 18:48:24.287753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:24.287351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl" event={"ID":"8db23ebd-f6db-4f11-b547-28ea2a57ec36","Type":"ContainerDied","Data":"c6036177821707208f25a70281b873fd60fb0ea213378d07cd87375838228338"}
Jul 31 18:48:25.416065 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.416041    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:25.558228 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.558142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle\") pod \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") "
Jul 31 18:48:25.558228 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.558203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util\") pod \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") "
Jul 31 18:48:25.558419 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.558239    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhrrm\" (UniqueName: \"kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm\") pod \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\" (UID: \"8db23ebd-f6db-4f11-b547-28ea2a57ec36\") "
Jul 31 18:48:25.558583 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.558553    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle" (OuterVolumeSpecName: "bundle") pod "8db23ebd-f6db-4f11-b547-28ea2a57ec36" (UID: "8db23ebd-f6db-4f11-b547-28ea2a57ec36"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:25.560384 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.560363    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm" (OuterVolumeSpecName: "kube-api-access-bhrrm") pod "8db23ebd-f6db-4f11-b547-28ea2a57ec36" (UID: "8db23ebd-f6db-4f11-b547-28ea2a57ec36"). InnerVolumeSpecName "kube-api-access-bhrrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:48:25.564993 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.564969    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util" (OuterVolumeSpecName: "util") pod "8db23ebd-f6db-4f11-b547-28ea2a57ec36" (UID: "8db23ebd-f6db-4f11-b547-28ea2a57ec36"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:25.659111 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.659082    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:25.659111 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.659107    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrrm\" (UniqueName: \"kubernetes.io/projected/8db23ebd-f6db-4f11-b547-28ea2a57ec36-kube-api-access-bhrrm\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:25.659111 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:25.659117    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db23ebd-f6db-4f11-b547-28ea2a57ec36-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:26.295830 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:26.295794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl" event={"ID":"8db23ebd-f6db-4f11-b547-28ea2a57ec36","Type":"ContainerDied","Data":"a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d"}
Jul 31 18:48:26.295975 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:26.295840    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftppcl"
Jul 31 18:48:26.295975 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:26.295844    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01df27430836060214d9d606cfcad18080349a75243b22e2e25fab58529b40d"
Jul 31 18:48:29.020434 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020401    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fkqfj"]
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020673    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="extract"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020685    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="extract"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020693    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="util"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020699    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="util"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020711    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="pull"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020716    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="pull"
Jul 31 18:48:29.020846 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.020765    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db23ebd-f6db-4f11-b547-28ea2a57ec36" containerName="extract"
Jul 31 18:48:29.024498 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.024482    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.027408 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.027379    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 18:48:29.028651 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.028632    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rjdch\""
Jul 31 18:48:29.028774 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.028653    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 18:48:29.037686 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.037661    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fkqfj"]
Jul 31 18:48:29.085037 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.085010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-bound-sa-token\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.085037 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.085042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85j5z\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-kube-api-access-85j5z\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.186585 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.186310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-bound-sa-token\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.186585 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.186365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85j5z\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-kube-api-access-85j5z\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.196632 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.196603    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-bound-sa-token\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.196737 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.196715    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85j5z\" (UniqueName: \"kubernetes.io/projected/168e40fa-428c-4d84-92cc-fb3c38e08230-kube-api-access-85j5z\") pod \"cert-manager-55b4d954d5-fkqfj\" (UID: \"168e40fa-428c-4d84-92cc-fb3c38e08230\") " pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.333261 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.333177    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fkqfj"
Jul 31 18:48:29.491969 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:29.491943    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fkqfj"]
Jul 31 18:48:29.494567 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:48:29.494541    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod168e40fa_428c_4d84_92cc_fb3c38e08230.slice/crio-17054b1d5b1674cd9607df83c35e00388fee97b3f8f66b135ef94104206a9e39 WatchSource:0}: Error finding container 17054b1d5b1674cd9607df83c35e00388fee97b3f8f66b135ef94104206a9e39: Status 404 returned error can't find the container with id 17054b1d5b1674cd9607df83c35e00388fee97b3f8f66b135ef94104206a9e39
Jul 31 18:48:30.311377 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:30.311340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fkqfj" event={"ID":"168e40fa-428c-4d84-92cc-fb3c38e08230","Type":"ContainerStarted","Data":"17054b1d5b1674cd9607df83c35e00388fee97b3f8f66b135ef94104206a9e39"}
Jul 31 18:48:32.320562 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:32.320482    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fkqfj" event={"ID":"168e40fa-428c-4d84-92cc-fb3c38e08230","Type":"ContainerStarted","Data":"fce7977113b7a1e095ea4d994074b246f82f20ebbbd895489c0a4019c3f099ca"}
Jul 31 18:48:32.338945 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:32.338874    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fkqfj" podStartSLOduration=0.888103588 podStartE2EDuration="3.338858171s" totalImagesPullingTime="2.450754583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:48:32.338087899 +0000 UTC m=+290.685669781" watchObservedRunningTime="2026-07-31 18:48:32.338858171 +0000 UTC m=+290.686440050"
Jul 31 18:48:42.076693 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:42.076663    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:48:42.079500 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:42.079476    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:48:42.088139 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:42.088120    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 18:48:44.604880 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.604844    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"]
Jul 31 18:48:44.608534 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.608512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.611348 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.611318    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 18:48:44.612616 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.612599    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 18:48:44.612705 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.612599    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cz2bg\""
Jul 31 18:48:44.620880 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.620850    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"]
Jul 31 18:48:44.703745 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.703705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.703745 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.703745    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.703962 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.703773    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fv6l\" (UniqueName: \"kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.804632 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.804596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.804632 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.804633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.804815 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.804658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fv6l\" (UniqueName: \"kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.804997 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.804978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.805033 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.805008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.814170 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.814139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fv6l\" (UniqueName: \"kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:44.917657 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:44.917565    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:45.051380 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:45.051323    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"]
Jul 31 18:48:45.053844 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:48:45.053819    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b281228_bfaa_406c_8bea_892fce91e660.slice/crio-84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c WatchSource:0}: Error finding container 84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c: Status 404 returned error can't find the container with id 84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c
Jul 31 18:48:45.367451 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:45.367404    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b281228-bfaa-406c-8bea-892fce91e660" containerID="abd973fd47080181050f41e547d04c968d55bf56f556a2938094afdd9133182b" exitCode=0
Jul 31 18:48:45.367611 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:45.367492    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r" event={"ID":"8b281228-bfaa-406c-8bea-892fce91e660","Type":"ContainerDied","Data":"abd973fd47080181050f41e547d04c968d55bf56f556a2938094afdd9133182b"}
Jul 31 18:48:45.367611 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:45.367541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r" event={"ID":"8b281228-bfaa-406c-8bea-892fce91e660","Type":"ContainerStarted","Data":"84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c"}
Jul 31 18:48:45.367954 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:45.367940    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 18:48:46.372992 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:46.372962    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b281228-bfaa-406c-8bea-892fce91e660" containerID="4dd18a90d7cb4d52cf64e2e2608bf0a972994a82d4fcf7b220a51b51154764e7" exitCode=0
Jul 31 18:48:46.373760 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:46.373026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r" event={"ID":"8b281228-bfaa-406c-8bea-892fce91e660","Type":"ContainerDied","Data":"4dd18a90d7cb4d52cf64e2e2608bf0a972994a82d4fcf7b220a51b51154764e7"}
Jul 31 18:48:47.379151 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:47.379111    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b281228-bfaa-406c-8bea-892fce91e660" containerID="77da6de1aedbd184d52c68973d73c0aa055987a430d2b62ce3c493e883d3184d" exitCode=0
Jul 31 18:48:47.379539 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:47.379204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r" event={"ID":"8b281228-bfaa-406c-8bea-892fce91e660","Type":"ContainerDied","Data":"77da6de1aedbd184d52c68973d73c0aa055987a430d2b62ce3c493e883d3184d"}
Jul 31 18:48:48.504420 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.504397    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:48.534329 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.534297    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util\") pod \"8b281228-bfaa-406c-8bea-892fce91e660\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") "
Jul 31 18:48:48.534532 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.534352    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fv6l\" (UniqueName: \"kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l\") pod \"8b281228-bfaa-406c-8bea-892fce91e660\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") "
Jul 31 18:48:48.534532 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.534382    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle\") pod \"8b281228-bfaa-406c-8bea-892fce91e660\" (UID: \"8b281228-bfaa-406c-8bea-892fce91e660\") "
Jul 31 18:48:48.535070 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.535045    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle" (OuterVolumeSpecName: "bundle") pod "8b281228-bfaa-406c-8bea-892fce91e660" (UID: "8b281228-bfaa-406c-8bea-892fce91e660"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:48.536679 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.536647    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l" (OuterVolumeSpecName: "kube-api-access-5fv6l") pod "8b281228-bfaa-406c-8bea-892fce91e660" (UID: "8b281228-bfaa-406c-8bea-892fce91e660"). InnerVolumeSpecName "kube-api-access-5fv6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:48:48.540255 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.540214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util" (OuterVolumeSpecName: "util") pod "8b281228-bfaa-406c-8bea-892fce91e660" (UID: "8b281228-bfaa-406c-8bea-892fce91e660"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:48:48.635431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.635319    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fv6l\" (UniqueName: \"kubernetes.io/projected/8b281228-bfaa-406c-8bea-892fce91e660-kube-api-access-5fv6l\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:48.635431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.635365    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:48.635431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:48.635380    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b281228-bfaa-406c-8bea-892fce91e660-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:48:49.387713 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:49.387677    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r" event={"ID":"8b281228-bfaa-406c-8bea-892fce91e660","Type":"ContainerDied","Data":"84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c"}
Jul 31 18:48:49.387880 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:49.387724    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a3a99ce9be952396b2fc18259ff4f825e8b2fa0d69629ae10d33a9823c507c"
Jul 31 18:48:49.387880 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:49.387690    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aff749r"
Jul 31 18:48:56.641253 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641217    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"]
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641494    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="pull"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641506    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="pull"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641516    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="extract"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641521    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="extract"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641528    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="util"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641534    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="util"
Jul 31 18:48:56.641659 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.641576    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b281228-bfaa-406c-8bea-892fce91e660" containerName="extract"
Jul 31 18:48:56.645310 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.645293    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.654338 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.654311    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6fdg\""
Jul 31 18:48:56.654485 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.654470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 31 18:48:56.654539 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.654516    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 31 18:48:56.654583 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.654559    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 31 18:48:56.655027 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.655012    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 31 18:48:56.664615 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.664591    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"]
Jul 31 18:48:56.693801 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.693746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.693971 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.693807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.693971 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.693874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbf42\" (UniqueName: \"kubernetes.io/projected/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-kube-api-access-wbf42\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.693971 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.693897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.693971 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.693926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.794686 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.794652    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.794877 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.794698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbf42\" (UniqueName: \"kubernetes.io/projected/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-kube-api-access-wbf42\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.794877 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.794717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.794877 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.794739    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.794877 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.794800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.795132 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.795109    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-manifests\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.797217 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.797184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-tmp\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.797348 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.797305    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.797406 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.797347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.814042 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.814008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbf42\" (UniqueName: \"kubernetes.io/projected/e7b6e52e-ff8b-4c80-8d30-524ca2e6707e-kube-api-access-wbf42\") pod \"opendatahub-operator-controller-manager-84c85c7ff9-vm4jm\" (UID: \"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e\") " pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:56.955151 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:56.955047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:48:57.105792 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:57.105761    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"]
Jul 31 18:48:57.108944 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:48:57.108915    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b6e52e_ff8b_4c80_8d30_524ca2e6707e.slice/crio-5747092333a8ca52adda77c071e93333cfd7f4a3ae643a461d53c6cd1327756d WatchSource:0}: Error finding container 5747092333a8ca52adda77c071e93333cfd7f4a3ae643a461d53c6cd1327756d: Status 404 returned error can't find the container with id 5747092333a8ca52adda77c071e93333cfd7f4a3ae643a461d53c6cd1327756d
Jul 31 18:48:57.419344 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:48:57.419311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm" event={"ID":"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e","Type":"ContainerStarted","Data":"5747092333a8ca52adda77c071e93333cfd7f4a3ae643a461d53c6cd1327756d"}
Jul 31 18:49:00.434513 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:00.434393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm" event={"ID":"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e","Type":"ContainerStarted","Data":"345481de7a6cc8af5df2155baee6eb85ae404783df7bca198721f470a37a5d4f"}
Jul 31 18:49:01.440240 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:01.440210    2557 generic.go:356] "Generic (PLEG): container finished" podID="e7b6e52e-ff8b-4c80-8d30-524ca2e6707e" containerID="345481de7a6cc8af5df2155baee6eb85ae404783df7bca198721f470a37a5d4f" exitCode=0
Jul 31 18:49:01.440759 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:01.440307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm" event={"ID":"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e","Type":"ContainerDied","Data":"345481de7a6cc8af5df2155baee6eb85ae404783df7bca198721f470a37a5d4f"}
Jul 31 18:49:02.446333 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:02.446294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm" event={"ID":"e7b6e52e-ff8b-4c80-8d30-524ca2e6707e","Type":"ContainerStarted","Data":"f3f3abab1ad1a678a32a702fd51eef6196bd241c50986007cb11faf0be131088"}
Jul 31 18:49:02.446756 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:02.446422    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:49:02.473428 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:02.473367    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm" podStartSLOduration=2.046510661 podStartE2EDuration="6.473345022s" totalImagesPullingTime="3.357855134s" totalInitContainerRuntime="1.068979227s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:48:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:02.471310659 +0000 UTC m=+320.818892539" watchObservedRunningTime="2026-07-31 18:49:02.473345022 +0000 UTC m=+320.820926903"
Jul 31 18:49:13.453017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:13.452990    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84c85c7ff9-vm4jm"
Jul 31 18:49:15.354671 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.354630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"]
Jul 31 18:49:15.362331 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.362299    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.366395 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.366367    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 18:49:15.366605 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.366579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 18:49:15.367670 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.367638    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dp8kl\""
Jul 31 18:49:15.367670 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.367654    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 18:49:15.367870 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.367656    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 18:49:15.367870 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.367655    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 18:49:15.375521 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.375498    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"]
Jul 31 18:49:15.441017 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.440983    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.441205 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.441024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-manager-config\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.441205 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.441049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.441205 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.441069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9k4\" (UniqueName: \"kubernetes.io/projected/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-kube-api-access-8b9k4\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.542208 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.542171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.542410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.542213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-manager-config\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.542410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.542248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.542410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.542319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9k4\" (UniqueName: \"kubernetes.io/projected/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-kube-api-access-8b9k4\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.543036 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.543017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-manager-config\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.544845 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.544824    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-metrics-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.544845 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.544837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-cert\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.557840 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.557819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9k4\" (UniqueName: \"kubernetes.io/projected/a6ae8e0a-d99d-446d-8cf8-d3822544c1d5-kube-api-access-8b9k4\") pod \"lws-controller-manager-6478f7cf66-pb29v\" (UID: \"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5\") " pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.672826 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.672744    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:15.828597 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:15.828570    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"]
Jul 31 18:49:15.830367 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:49:15.830343    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ae8e0a_d99d_446d_8cf8_d3822544c1d5.slice/crio-7d12bcdd9cd66ddb62848cbcaef247f706ce87007df6a351adceec3826c791d5 WatchSource:0}: Error finding container 7d12bcdd9cd66ddb62848cbcaef247f706ce87007df6a351adceec3826c791d5: Status 404 returned error can't find the container with id 7d12bcdd9cd66ddb62848cbcaef247f706ce87007df6a351adceec3826c791d5
Jul 31 18:49:16.508524 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:16.508491    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v" event={"ID":"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5","Type":"ContainerStarted","Data":"7d12bcdd9cd66ddb62848cbcaef247f706ce87007df6a351adceec3826c791d5"}
Jul 31 18:49:18.520648 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:18.520609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v" event={"ID":"a6ae8e0a-d99d-446d-8cf8-d3822544c1d5","Type":"ContainerStarted","Data":"ea41c51b327af248c83378235aeeae6bb5b59d02130ab4967bee555174a12124"}
Jul 31 18:49:18.521038 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:18.520681    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:49:18.538069 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:18.538019    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v" podStartSLOduration=1.5416467059999999 podStartE2EDuration="3.538004847s" totalImagesPullingTime="1.996358141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:18.536886687 +0000 UTC m=+336.884468568" watchObservedRunningTime="2026-07-31 18:49:18.538004847 +0000 UTC m=+336.885586771"
Jul 31 18:49:23.873175 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.873136    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"]
Jul 31 18:49:23.876517 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.876502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:23.882405 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882383    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 18:49:23.882560 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 31 18:49:23.882560 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882489    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 18:49:23.882946 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882676    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 31 18:49:23.882946 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882818    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qtxwn\""
Jul 31 18:49:23.882946 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.882828    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 18:49:23.883635 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.883619    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 31 18:49:23.888756 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:23.888738    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"]
Jul 31 18:49:24.012587 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012587 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxbr\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-kube-api-access-qvxbr\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c164c810-acc7-48b5-b306-f7d1ceec06fd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012768    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.012866 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.012804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114160 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxbr\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-kube-api-access-qvxbr\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114365 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114308    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c164c810-acc7-48b5-b306-f7d1ceec06fd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.114985 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.114958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.116776 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.116754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c164c810-acc7-48b5-b306-f7d1ceec06fd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.116927 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.116903    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.117119 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.117104    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.117182 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.117145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.126778 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.126726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.126859 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.126842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxbr\" (UniqueName: \"kubernetes.io/projected/c164c810-acc7-48b5-b306-f7d1ceec06fd-kube-api-access-qvxbr\") pod \"istiod-openshift-gateway-5677ccb49b-88mdk\" (UID: \"c164c810-acc7-48b5-b306-f7d1ceec06fd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.186299 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.186260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:24.329750 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.329724    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"]
Jul 31 18:49:24.331584 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:49:24.331558    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc164c810_acc7_48b5_b306_f7d1ceec06fd.slice/crio-16fb159d3b682e659f563df0caaf2ef368ccba0dd105203332d5b483a3afce08 WatchSource:0}: Error finding container 16fb159d3b682e659f563df0caaf2ef368ccba0dd105203332d5b483a3afce08: Status 404 returned error can't find the container with id 16fb159d3b682e659f563df0caaf2ef368ccba0dd105203332d5b483a3afce08
Jul 31 18:49:24.547217 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:24.547123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk" event={"ID":"c164c810-acc7-48b5-b306-f7d1ceec06fd","Type":"ContainerStarted","Data":"16fb159d3b682e659f563df0caaf2ef368ccba0dd105203332d5b483a3afce08"}
Jul 31 18:49:27.068859 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:27.068827    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 31 18:49:27.069101 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:27.068899    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 31 18:49:27.564759 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:27.564721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk" event={"ID":"c164c810-acc7-48b5-b306-f7d1ceec06fd","Type":"ContainerStarted","Data":"962536fe3701034a1dd0640bd90e0fd764fd2f6a2626dae86845eda5fffabd12"}
Jul 31 18:49:27.565093 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:27.565041    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:27.590264 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:27.590206    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk" podStartSLOduration=1.855188447 podStartE2EDuration="4.590185811s" totalImagesPullingTime="2.734997364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:49:27.58905957 +0000 UTC m=+345.936641459" watchObservedRunningTime="2026-07-31 18:49:27.590185811 +0000 UTC m=+345.937767691"
Jul 31 18:49:28.573540 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:28.573511    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-88mdk"
Jul 31 18:49:29.527733 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:49:29.527707    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6478f7cf66-pb29v"
Jul 31 18:50:20.922759 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.922725    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"]
Jul 31 18:50:20.925134 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.925109    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:20.927820 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.927800    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 18:50:20.928852 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.928835    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 18:50:20.928943 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.928852    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cz2bg\""
Jul 31 18:50:20.935584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.935533    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"]
Jul 31 18:50:20.971884 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.971861    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:20.972059 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.971921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:20.972059 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.971982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblpr\" (UniqueName: \"kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:20.997615 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.997586    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"]
Jul 31 18:50:20.999775 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:20.999761    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.010619 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.010594    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"]
Jul 31 18:50:21.073309 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.073517 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblpr\" (UniqueName: \"kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.073517 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.073642 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.073642 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.073642 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073609    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.073834 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.073867 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.073847    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.084401 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.084377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblpr\" (UniqueName: \"kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.095724 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.095703    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"]
Jul 31 18:50:21.097817 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.097804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.108942 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.108919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"]
Jul 31 18:50:21.174165 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.174165 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.174165 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.174396 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174229    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.174396 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.174396 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbx2z\" (UniqueName: \"kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.174537 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174518    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.174576 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.174541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.183369 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.183348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.197344 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.197315    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"]
Jul 31 18:50:21.199585 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.199571    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.213120 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.213094    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"]
Jul 31 18:50:21.233885 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.233857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:21.274979 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.274950    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vnx\" (UniqueName: \"kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.275135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.274998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.275135 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275069    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.275225 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.275225 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbx2z\" (UniqueName: \"kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.275327 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.275431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275410    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.275531 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.275510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.286923 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.286899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbx2z\" (UniqueName: \"kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.308507 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.308480    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:21.377730 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.377341    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"]
Jul 31 18:50:21.377977 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.377946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.378066 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.377987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vnx\" (UniqueName: \"kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.378066 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.378022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.378509 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.378427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.378619 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.378522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.382754 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:50:21.382701    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe3f805_94b8_47df_89dd_096dd12aa6a3.slice/crio-b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f WatchSource:0}: Error finding container b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f: Status 404 returned error can't find the container with id b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f
Jul 31 18:50:21.388985 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.388959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vnx\" (UniqueName: \"kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.407141 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.406693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:21.471711 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.471639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"]
Jul 31 18:50:21.472810 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:50:21.472781    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f3b0f7_3764_4739_bf12_caf7848b8119.slice/crio-a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada WatchSource:0}: Error finding container a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada: Status 404 returned error can't find the container with id a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada
Jul 31 18:50:21.508260 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.508204    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:21.561538 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.561503    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"]
Jul 31 18:50:21.573701 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:50:21.573660    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ceee2d_0654_4411_9f0d_7c63a3696843.slice/crio-bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64 WatchSource:0}: Error finding container bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64: Status 404 returned error can't find the container with id bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64
Jul 31 18:50:21.666613 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.666588    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"]
Jul 31 18:50:21.682255 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:50:21.682229    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc1a59a_3f61_4962_9d33_cc31ea830a4e.slice/crio-12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a WatchSource:0}: Error finding container 12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a: Status 404 returned error can't find the container with id 12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a
Jul 31 18:50:21.803108 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.803066    2557 generic.go:356] "Generic (PLEG): container finished" podID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerID="eb12688d0600a18e29abbdcef0b21e026ad25cacafb089f9a1903338e86661bd" exitCode=0
Jul 31 18:50:21.803244 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.803142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2" event={"ID":"63ceee2d-0654-4411-9f0d-7c63a3696843","Type":"ContainerDied","Data":"eb12688d0600a18e29abbdcef0b21e026ad25cacafb089f9a1903338e86661bd"}
Jul 31 18:50:21.803244 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.803196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2" event={"ID":"63ceee2d-0654-4411-9f0d-7c63a3696843","Type":"ContainerStarted","Data":"bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64"}
Jul 31 18:50:21.804924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.804842    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerStarted","Data":"609ecc31304b9fedafc73c32ba3e5c3add67c706d02bc6f124fb1e9a610734a1"}
Jul 31 18:50:21.804924 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.804896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerStarted","Data":"12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a"}
Jul 31 18:50:21.806421 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.806399    2557 generic.go:356] "Generic (PLEG): container finished" podID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerID="549c91aae1de0cfb94e06534604c23b3a4fe54fd4eed4ccd75e11e639d981a6c" exitCode=0
Jul 31 18:50:21.806545 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.806474    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl" event={"ID":"92f3b0f7-3764-4739-bf12-caf7848b8119","Type":"ContainerDied","Data":"549c91aae1de0cfb94e06534604c23b3a4fe54fd4eed4ccd75e11e639d981a6c"}
Jul 31 18:50:21.806545 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.806522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl" event={"ID":"92f3b0f7-3764-4739-bf12-caf7848b8119","Type":"ContainerStarted","Data":"a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada"}
Jul 31 18:50:21.808331 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.808313    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerID="c0a08835836dee287f8b1b6d0a18eb9e02d55bf531e2ce82936a14973384b3ad" exitCode=0
Jul 31 18:50:21.808410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.808343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn" event={"ID":"fbe3f805-94b8-47df-89dd-096dd12aa6a3","Type":"ContainerDied","Data":"c0a08835836dee287f8b1b6d0a18eb9e02d55bf531e2ce82936a14973384b3ad"}
Jul 31 18:50:21.808410 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:21.808368    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn" event={"ID":"fbe3f805-94b8-47df-89dd-096dd12aa6a3","Type":"ContainerStarted","Data":"b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f"}
Jul 31 18:50:22.813882 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:22.813847    2557 generic.go:356] "Generic (PLEG): container finished" podID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerID="609ecc31304b9fedafc73c32ba3e5c3add67c706d02bc6f124fb1e9a610734a1" exitCode=0
Jul 31 18:50:22.814246 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:22.813898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerDied","Data":"609ecc31304b9fedafc73c32ba3e5c3add67c706d02bc6f124fb1e9a610734a1"}
Jul 31 18:50:23.820677 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.820636    2557 generic.go:356] "Generic (PLEG): container finished" podID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerID="94c33adff033abeb48bb1eb38fa3ec8f0e2507872e3b46652d041f6dba93c953" exitCode=0
Jul 31 18:50:23.821100 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.820713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl" event={"ID":"92f3b0f7-3764-4739-bf12-caf7848b8119","Type":"ContainerDied","Data":"94c33adff033abeb48bb1eb38fa3ec8f0e2507872e3b46652d041f6dba93c953"}
Jul 31 18:50:23.822604 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.822580    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerID="5c0d711c01d267a0abbbfb10a65b70203ce41407e33d4602ce5d77f7d109b78b" exitCode=0
Jul 31 18:50:23.822684 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.822659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn" event={"ID":"fbe3f805-94b8-47df-89dd-096dd12aa6a3","Type":"ContainerDied","Data":"5c0d711c01d267a0abbbfb10a65b70203ce41407e33d4602ce5d77f7d109b78b"}
Jul 31 18:50:23.824465 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.824426    2557 generic.go:356] "Generic (PLEG): container finished" podID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerID="49ef81e3e06d8944b8d076d242b2b62e77e85ac41fd2f24307497c1a0a3de684" exitCode=0
Jul 31 18:50:23.824592 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.824464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2" event={"ID":"63ceee2d-0654-4411-9f0d-7c63a3696843","Type":"ContainerDied","Data":"49ef81e3e06d8944b8d076d242b2b62e77e85ac41fd2f24307497c1a0a3de684"}
Jul 31 18:50:23.826487 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.826469    2557 generic.go:356] "Generic (PLEG): container finished" podID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerID="896e7c9e051f040249bf0fbcdf80bd0dcba315fc5b437c7c8180a2c82f0c6dca" exitCode=0
Jul 31 18:50:23.826550 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:23.826509    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerDied","Data":"896e7c9e051f040249bf0fbcdf80bd0dcba315fc5b437c7c8180a2c82f0c6dca"}
Jul 31 18:50:24.833256 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.833221    2557 generic.go:356] "Generic (PLEG): container finished" podID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerID="65cc20b87e878166d4031b512be36652cfb6ef3d2c407b536625e511a0c32890" exitCode=0
Jul 31 18:50:24.833716 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.833317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2" event={"ID":"63ceee2d-0654-4411-9f0d-7c63a3696843","Type":"ContainerDied","Data":"65cc20b87e878166d4031b512be36652cfb6ef3d2c407b536625e511a0c32890"}
Jul 31 18:50:24.835290 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.835264    2557 generic.go:356] "Generic (PLEG): container finished" podID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerID="d4be90fa0f32ecccc200632a2bfec6a3f1eb2d71dba9dad9849c62b0b442254b" exitCode=0
Jul 31 18:50:24.835401 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.835348    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerDied","Data":"d4be90fa0f32ecccc200632a2bfec6a3f1eb2d71dba9dad9849c62b0b442254b"}
Jul 31 18:50:24.837290 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.837271    2557 generic.go:356] "Generic (PLEG): container finished" podID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerID="d393c5cdc12c23d6a2ec24754d88e9bdc4c3a98a764e9c5fe194fb657bf28de1" exitCode=0
Jul 31 18:50:24.837377 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.837351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl" event={"ID":"92f3b0f7-3764-4739-bf12-caf7848b8119","Type":"ContainerDied","Data":"d393c5cdc12c23d6a2ec24754d88e9bdc4c3a98a764e9c5fe194fb657bf28de1"}
Jul 31 18:50:24.839482 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.839459    2557 generic.go:356] "Generic (PLEG): container finished" podID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerID="08f85be0c38e153f56830ba7f54f7fea936ce94d363659b2a6bbf0d87370460e" exitCode=0
Jul 31 18:50:24.839574 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:24.839479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn" event={"ID":"fbe3f805-94b8-47df-89dd-096dd12aa6a3","Type":"ContainerDied","Data":"08f85be0c38e153f56830ba7f54f7fea936ce94d363659b2a6bbf0d87370460e"}
Jul 31 18:50:26.034356 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.034330    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:26.037577 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.037557    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:50:26.057120 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.057093    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:26.061840 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.061819    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:26.123022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.122929    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle\") pod \"63ceee2d-0654-4411-9f0d-7c63a3696843\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") "
Jul 31 18:50:26.123022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.122987    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util\") pod \"92f3b0f7-3764-4739-bf12-caf7848b8119\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") "
Jul 31 18:50:26.123022 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle\") pod \"92f3b0f7-3764-4739-bf12-caf7848b8119\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") "
Jul 31 18:50:26.123242 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util\") pod \"63ceee2d-0654-4411-9f0d-7c63a3696843\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") "
Jul 31 18:50:26.123242 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util\") pod \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") "
Jul 31 18:50:26.123242 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123071    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9\") pod \"92f3b0f7-3764-4739-bf12-caf7848b8119\" (UID: \"92f3b0f7-3764-4739-bf12-caf7848b8119\") "
Jul 31 18:50:26.123242 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123095    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fblpr\" (UniqueName: \"kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr\") pod \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") "
Jul 31 18:50:26.123242 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle\") pod \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\" (UID: \"fbe3f805-94b8-47df-89dd-096dd12aa6a3\") "
Jul 31 18:50:26.123465 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123370    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbx2z\" (UniqueName: \"kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z\") pod \"63ceee2d-0654-4411-9f0d-7c63a3696843\" (UID: \"63ceee2d-0654-4411-9f0d-7c63a3696843\") "
Jul 31 18:50:26.123465 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123416    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle\") pod \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") "
Jul 31 18:50:26.123581 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123482    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6vnx\" (UniqueName: \"kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx\") pod \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") "
Jul 31 18:50:26.123581 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123515    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util\") pod \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\" (UID: \"bbc1a59a-3f61-4962-9d33-cc31ea830a4e\") "
Jul 31 18:50:26.123731 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123703    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle" (OuterVolumeSpecName: "bundle") pod "fbe3f805-94b8-47df-89dd-096dd12aa6a3" (UID: "fbe3f805-94b8-47df-89dd-096dd12aa6a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.123914 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.123796    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle" (OuterVolumeSpecName: "bundle") pod "63ceee2d-0654-4411-9f0d-7c63a3696843" (UID: "63ceee2d-0654-4411-9f0d-7c63a3696843"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.124548 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.124520    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle" (OuterVolumeSpecName: "bundle") pod "bbc1a59a-3f61-4962-9d33-cc31ea830a4e" (UID: "bbc1a59a-3f61-4962-9d33-cc31ea830a4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.126786 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.126756    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle" (OuterVolumeSpecName: "bundle") pod "92f3b0f7-3764-4739-bf12-caf7848b8119" (UID: "92f3b0f7-3764-4739-bf12-caf7848b8119"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.130262 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.130241    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util" (OuterVolumeSpecName: "util") pod "bbc1a59a-3f61-4962-9d33-cc31ea830a4e" (UID: "bbc1a59a-3f61-4962-9d33-cc31ea830a4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.131812 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.131789    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util" (OuterVolumeSpecName: "util") pod "fbe3f805-94b8-47df-89dd-096dd12aa6a3" (UID: "fbe3f805-94b8-47df-89dd-096dd12aa6a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.131920 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.131902    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx" (OuterVolumeSpecName: "kube-api-access-v6vnx") pod "bbc1a59a-3f61-4962-9d33-cc31ea830a4e" (UID: "bbc1a59a-3f61-4962-9d33-cc31ea830a4e"). InnerVolumeSpecName "kube-api-access-v6vnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:50:26.131920 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.131913    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9" (OuterVolumeSpecName: "kube-api-access-5ljt9") pod "92f3b0f7-3764-4739-bf12-caf7848b8119" (UID: "92f3b0f7-3764-4739-bf12-caf7848b8119"). InnerVolumeSpecName "kube-api-access-5ljt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:50:26.132014 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.131922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z" (OuterVolumeSpecName: "kube-api-access-mbx2z") pod "63ceee2d-0654-4411-9f0d-7c63a3696843" (UID: "63ceee2d-0654-4411-9f0d-7c63a3696843"). InnerVolumeSpecName "kube-api-access-mbx2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:50:26.132888 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.132845    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util" (OuterVolumeSpecName: "util") pod "63ceee2d-0654-4411-9f0d-7c63a3696843" (UID: "63ceee2d-0654-4411-9f0d-7c63a3696843"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.133729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.133703    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util" (OuterVolumeSpecName: "util") pod "92f3b0f7-3764-4739-bf12-caf7848b8119" (UID: "92f3b0f7-3764-4739-bf12-caf7848b8119"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 18:50:26.133729 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.133709    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr" (OuterVolumeSpecName: "kube-api-access-fblpr") pod "fbe3f805-94b8-47df-89dd-096dd12aa6a3" (UID: "fbe3f805-94b8-47df-89dd-096dd12aa6a3"). InnerVolumeSpecName "kube-api-access-fblpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 18:50:26.224753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224721    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.224753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224751    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbx2z\" (UniqueName: \"kubernetes.io/projected/63ceee2d-0654-4411-9f0d-7c63a3696843-kube-api-access-mbx2z\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.224753 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224761    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224770    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vnx\" (UniqueName: \"kubernetes.io/projected/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-kube-api-access-v6vnx\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224779    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbc1a59a-3f61-4962-9d33-cc31ea830a4e-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224786    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224795    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224802    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92f3b0f7-3764-4739-bf12-caf7848b8119-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224809    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ceee2d-0654-4411-9f0d-7c63a3696843-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224815    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbe3f805-94b8-47df-89dd-096dd12aa6a3-util\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224823    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljt9\" (UniqueName: \"kubernetes.io/projected/92f3b0f7-3764-4739-bf12-caf7848b8119-kube-api-access-5ljt9\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.225054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.224831    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblpr\" (UniqueName: \"kubernetes.io/projected/fbe3f805-94b8-47df-89dd-096dd12aa6a3-kube-api-access-fblpr\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 18:50:26.854407 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.854380    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2"
Jul 31 18:50:26.854407 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.854401    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rdtk2" event={"ID":"63ceee2d-0654-4411-9f0d-7c63a3696843","Type":"ContainerDied","Data":"bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64"}
Jul 31 18:50:26.854765 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.854474    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae01d52accd940344b4c21b02270fe7e54a1aa6c0eaf72359bc5660e2f8db64"
Jul 31 18:50:26.856468 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.856401    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs"
Jul 31 18:50:26.856468 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.856428    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07qnxs" event={"ID":"bbc1a59a-3f61-4962-9d33-cc31ea830a4e","Type":"ContainerDied","Data":"12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a"}
Jul 31 18:50:26.856704 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.856494    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12530b61b48cd295d667ec4ef7d457193a2fe293695e44ff9cbcd21cc938a47a"
Jul 31 18:50:26.858457 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.858412    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl" event={"ID":"92f3b0f7-3764-4739-bf12-caf7848b8119","Type":"ContainerDied","Data":"a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada"}
Jul 31 18:50:26.858586 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.858476    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1436a52bcf0dfc3a1e051b03fe13db61febf7e6ae4ca0bb9547aa651e88cada"
Jul 31 18:50:26.858586 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.858425    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfj5vl"
Jul 31 18:50:26.860296 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.860277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn" event={"ID":"fbe3f805-94b8-47df-89dd-096dd12aa6a3","Type":"ContainerDied","Data":"b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f"}
Jul 31 18:50:26.860415 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.860311    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b852567978b087162f84b16680a5e997609686f5950f016c6f7b2d4f356c7a2f"
Jul 31 18:50:26.860415 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:50:26.860370    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clq7nn"
Jul 31 18:51:01.573692 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.573658    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"]
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.573983    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.573995    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574005    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574010    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574016    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574021    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574031    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574036    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574042    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574047    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574054    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574059    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574065    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574075    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574079    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="extract"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574086    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574090    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="pull"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574097    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574102    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574107    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="util"
Jul 31 18:51:01.574103 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574112    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="util"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574117    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="extract"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574122    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="extract"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574160    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f3b0f7-3764-4739-bf12-caf7848b8119" containerName="extract"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574166    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc1a59a-3f61-4962-9d33-cc31ea830a4e" containerName="extract"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574171    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe3f805-94b8-47df-89dd-096dd12aa6a3" containerName="extract"
Jul 31 18:51:01.574796 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.574179    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ceee2d-0654-4411-9f0d-7c63a3696843" containerName="extract"
Jul 31 18:51:01.578473 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.578433    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:01.581997 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.581971    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9gswt\""
Jul 31 18:51:01.583112 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.583093    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 18:51:01.583245 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.583119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 18:51:01.595616 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.595596    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"]
Jul 31 18:51:01.707042 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.707009    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnx9\" (UniqueName: \"kubernetes.io/projected/0cae623c-0f33-4820-8917-a68ffd41eeb1-kube-api-access-6gnx9\") pod \"limitador-operator-controller-manager-6c549b6d85-kftfz\" (UID: \"0cae623c-0f33-4820-8917-a68ffd41eeb1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:01.808310 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.808217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnx9\" (UniqueName: \"kubernetes.io/projected/0cae623c-0f33-4820-8917-a68ffd41eeb1-kube-api-access-6gnx9\") pod \"limitador-operator-controller-manager-6c549b6d85-kftfz\" (UID: \"0cae623c-0f33-4820-8917-a68ffd41eeb1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:01.819149 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.819120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnx9\" (UniqueName: \"kubernetes.io/projected/0cae623c-0f33-4820-8917-a68ffd41eeb1-kube-api-access-6gnx9\") pod \"limitador-operator-controller-manager-6c549b6d85-kftfz\" (UID: \"0cae623c-0f33-4820-8917-a68ffd41eeb1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:01.889248 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:01.889220    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:02.024567 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:02.024538    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"]
Jul 31 18:51:02.027054 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:51:02.027028    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cae623c_0f33_4820_8917_a68ffd41eeb1.slice/crio-0971cf5462e020c532e0391b862ee6e328280f416d3d4d27c481a093c3f517db WatchSource:0}: Error finding container 0971cf5462e020c532e0391b862ee6e328280f416d3d4d27c481a093c3f517db: Status 404 returned error can't find the container with id 0971cf5462e020c532e0391b862ee6e328280f416d3d4d27c481a093c3f517db
Jul 31 18:51:03.027816 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:03.027767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz" event={"ID":"0cae623c-0f33-4820-8917-a68ffd41eeb1","Type":"ContainerStarted","Data":"0971cf5462e020c532e0391b862ee6e328280f416d3d4d27c481a093c3f517db"}
Jul 31 18:51:05.039272 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:05.039234    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz" event={"ID":"0cae623c-0f33-4820-8917-a68ffd41eeb1","Type":"ContainerStarted","Data":"8e48245bacfb0e78898fcc229a460058f744f6d0035885bf1624782217cffcfe"}
Jul 31 18:51:05.039682 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:05.039292    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:51:05.062868 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:05.062825    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz" podStartSLOduration=1.714546425 podStartE2EDuration="4.062809939s" totalImagesPullingTime="2.348263514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:51:05.061391926 +0000 UTC m=+443.408973810" watchObservedRunningTime="2026-07-31 18:51:05.062809939 +0000 UTC m=+443.410391819"
Jul 31 18:51:16.046507 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:51:16.046476    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-kftfz"
Jul 31 18:53:17.679755 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.679723    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"]
Jul 31 18:53:17.681807 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.681790    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.684577 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.684553    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 31 18:53:17.684680 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.684562    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 31 18:53:17.685638 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.685616    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs6tj\""
Jul 31 18:53:17.685725 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.685648    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 31 18:53:17.696838 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.696814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"]
Jul 31 18:53:17.834175 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.834351 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d93fcb4-dd4d-4628-827e-84182484c5ce-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.834351 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.834351 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834270    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.834351 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rst\" (UniqueName: \"kubernetes.io/projected/7d93fcb4-dd4d-4628-827e-84182484c5ce-kube-api-access-m9rst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.834351 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.834339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.934943 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.934869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.934943 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.934912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.934943 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.934932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rst\" (UniqueName: \"kubernetes.io/projected/7d93fcb4-dd4d-4628-827e-84182484c5ce-kube-api-access-m9rst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935144 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.934974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935144 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.935017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935144 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.935068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d93fcb4-dd4d-4628-827e-84182484c5ce-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935296 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.935274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935431 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.935403    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.935553 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.935484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.937330 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.937304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d93fcb4-dd4d-4628-827e-84182484c5ce-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.937598 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.937580    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d93fcb4-dd4d-4628-827e-84182484c5ce-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.944743 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.944718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rst\" (UniqueName: \"kubernetes.io/projected/7d93fcb4-dd4d-4628-827e-84182484c5ce-kube-api-access-m9rst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj\" (UID: \"7d93fcb4-dd4d-4628-827e-84182484c5ce\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:17.993804 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:17.993747    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:18.134637 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:18.134611    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"]
Jul 31 18:53:18.136403 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:53:18.136374    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d93fcb4_dd4d_4628_827e_84182484c5ce.slice/crio-83d3d838801855d4b47a4f239312b1ba5fcb85e35ada800cf83d0d7121b51957 WatchSource:0}: Error finding container 83d3d838801855d4b47a4f239312b1ba5fcb85e35ada800cf83d0d7121b51957: Status 404 returned error can't find the container with id 83d3d838801855d4b47a4f239312b1ba5fcb85e35ada800cf83d0d7121b51957
Jul 31 18:53:18.638128 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:18.638093    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj" event={"ID":"7d93fcb4-dd4d-4628-827e-84182484c5ce","Type":"ContainerStarted","Data":"83d3d838801855d4b47a4f239312b1ba5fcb85e35ada800cf83d0d7121b51957"}
Jul 31 18:53:19.644160 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:19.644117    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj" event={"ID":"7d93fcb4-dd4d-4628-827e-84182484c5ce","Type":"ContainerStarted","Data":"d03152368398be628eddb440ffc083ceb75220e954575f7bfc3ab91ffed6568b"}
Jul 31 18:53:19.664714 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:19.664663    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj" podStartSLOduration=1.6771975220000002 podStartE2EDuration="2.664642486s" totalImagesPullingTime="987.444964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:53:19.663226187 +0000 UTC m=+578.010808066" watchObservedRunningTime="2026-07-31 18:53:19.664642486 +0000 UTC m=+578.012224365"
Jul 31 18:53:27.994264 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:27.994229    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:27.994692 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:27.994302    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:28.006545 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:28.006519    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:28.696479 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:28.696431    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj"
Jul 31 18:53:42.108681 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:42.108650    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:53:42.109121 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:53:42.108876    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:54:10.599167 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.599088    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"]
Jul 31 18:54:10.602378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.602363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.605080 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.605058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 31 18:54:10.611624 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.611597    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"]
Jul 31 18:54:10.633391 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.633584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.633584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.633584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.633584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4bd\" (UniqueName: \"kubernetes.io/projected/2d0ad837-6657-47bf-9fd7-789210d97331-kube-api-access-2p4bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.633584 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.633522    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad837-6657-47bf-9fd7-789210d97331-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734380 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734380 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734657 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734657 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734425    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734657 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4bd\" (UniqueName: \"kubernetes.io/projected/2d0ad837-6657-47bf-9fd7-789210d97331-kube-api-access-2p4bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734657 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad837-6657-47bf-9fd7-789210d97331-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734863 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734919 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734895    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.734976 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.734927    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.736913 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.736895    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0ad837-6657-47bf-9fd7-789210d97331-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.737054 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.737040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0ad837-6657-47bf-9fd7-789210d97331-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.744313 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.744294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4bd\" (UniqueName: \"kubernetes.io/projected/2d0ad837-6657-47bf-9fd7-789210d97331-kube-api-access-2p4bd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5825\" (UID: \"2d0ad837-6657-47bf-9fd7-789210d97331\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:10.912477 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:10.912365    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:11.043167 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:11.043140    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"]
Jul 31 18:54:11.044681 ip-10-0-128-163 kubenswrapper[2557]: W0731 18:54:11.044653    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0ad837_6657_47bf_9fd7_789210d97331.slice/crio-8f98446a03ac20e0b3452d4ce0e44aa96b7d910ceb35e104d7ce3ea3b0ce3ee8 WatchSource:0}: Error finding container 8f98446a03ac20e0b3452d4ce0e44aa96b7d910ceb35e104d7ce3ea3b0ce3ee8: Status 404 returned error can't find the container with id 8f98446a03ac20e0b3452d4ce0e44aa96b7d910ceb35e104d7ce3ea3b0ce3ee8
Jul 31 18:54:11.045998 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:11.045979    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 18:54:11.891001 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:11.890968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825" event={"ID":"2d0ad837-6657-47bf-9fd7-789210d97331","Type":"ContainerStarted","Data":"494960f3e97890222a0f2cf231795f3c5a93b041df0fdb588001f836d6ef7745"}
Jul 31 18:54:11.891378 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:11.891020    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825" event={"ID":"2d0ad837-6657-47bf-9fd7-789210d97331","Type":"ContainerStarted","Data":"8f98446a03ac20e0b3452d4ce0e44aa96b7d910ceb35e104d7ce3ea3b0ce3ee8"}
Jul 31 18:54:11.911939 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:11.911887    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825" podStartSLOduration=1.7309516569999999 podStartE2EDuration="1.911872112s" totalImagesPullingTime="180.920455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 18:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 18:54:11.910094044 +0000 UTC m=+630.257675925" watchObservedRunningTime="2026-07-31 18:54:11.911872112 +0000 UTC m=+630.259453991"
Jul 31 18:54:20.913121 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:20.913084    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:20.913621 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:20.913158    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:20.925769 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:20.925745    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:54:20.943246 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:54:20.943214    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5825"
Jul 31 18:58:42.142502 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:58:42.142409    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 18:58:42.143922 ip-10-0-128-163 kubenswrapper[2557]: I0731 18:58:42.143903    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:03:42.171259 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:03:42.171230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:03:42.175320 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:03:42.175299    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:08:42.205840 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:08:42.205814    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:08:42.207911 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:08:42.207893    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:12:41.500621 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.500580    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:12:41.503776 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.503756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.506376 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.506348    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b30c0866-dockercfg-4rld4\""
Jul 31 19:12:41.506531 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.506380    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b30c0866\""
Jul 31 19:12:41.513693 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.513663    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:12:41.652255 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.652219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75s7\" (UniqueName: \"kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.652423 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.652278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.753403 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.753311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75s7\" (UniqueName: \"kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.753403 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.753377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.754053 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.754035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.762576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.762553    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75s7\" (UniqueName: \"kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7\") pod \"payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.816028 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.815991    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:41.946623 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.946595    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:12:41.948492 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:12:41.948461    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb175d110_8feb_4a78_8c41_c38a7922f3bb.slice/crio-3acd3d909cc9d76a6a16a3c9ddf5da104fe5900d3b851e4f468d471ea6dad3fa WatchSource:0}: Error finding container 3acd3d909cc9d76a6a16a3c9ddf5da104fe5900d3b851e4f468d471ea6dad3fa: Status 404 returned error can't find the container with id 3acd3d909cc9d76a6a16a3c9ddf5da104fe5900d3b851e4f468d471ea6dad3fa
Jul 31 19:12:41.950344 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:41.950325    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 19:12:42.961672 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:42.961630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" event={"ID":"b175d110-8feb-4a78-8c41-c38a7922f3bb","Type":"ContainerStarted","Data":"3acd3d909cc9d76a6a16a3c9ddf5da104fe5900d3b851e4f468d471ea6dad3fa"}
Jul 31 19:12:43.969595 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:43.969514    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" event={"ID":"b175d110-8feb-4a78-8c41-c38a7922f3bb","Type":"ContainerStarted","Data":"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"}
Jul 31 19:12:43.969964 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:43.969662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:12:43.989078 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:43.989024    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" podStartSLOduration=1.322715207 podStartE2EDuration="2.989010785s" totalImagesPullingTime="1.666295578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:12:43.986076097 +0000 UTC m=+1742.333657976" watchObservedRunningTime="2026-07-31 19:12:43.989010785 +0000 UTC m=+1742.336592665"
Jul 31 19:12:54.977551 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:12:54.977522    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:13:10.414474 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.414420    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"]
Jul 31 19:13:10.418759 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.418742    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:10.421434 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.421411    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:13:10.421592 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.421470    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:13:10.426184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.425925    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"]
Jul 31 19:13:10.485607 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.485575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b30c0866-lrntj\" (UID: \"42c625ca-30c9-447f-9268-3aac2f870842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:10.586926 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.586892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b30c0866-lrntj\" (UID: \"42c625ca-30c9-447f-9268-3aac2f870842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:10.587552 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.587534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b30c0866-lrntj\" (UID: \"42c625ca-30c9-447f-9268-3aac2f870842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:10.729379 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.729287    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:10.859388 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:10.859363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"]
Jul 31 19:13:10.861907 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:13:10.861874    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c625ca_30c9_447f_9268_3aac2f870842.slice/crio-ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8 WatchSource:0}: Error finding container ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8: Status 404 returned error can't find the container with id ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8
Jul 31 19:13:11.107532 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:11.107496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj" event={"ID":"42c625ca-30c9-447f-9268-3aac2f870842","Type":"ContainerStarted","Data":"ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8"}
Jul 31 19:13:15.131838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:15.131802    2557 generic.go:356] "Generic (PLEG): container finished" podID="42c625ca-30c9-447f-9268-3aac2f870842" containerID="709bc03c3978912eea93efe3e5ced6767cafc69fa278ef5025213b5bc44c5809" exitCode=0
Jul 31 19:13:15.132237 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:15.131887    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj" event={"ID":"42c625ca-30c9-447f-9268-3aac2f870842","Type":"ContainerDied","Data":"709bc03c3978912eea93efe3e5ced6767cafc69fa278ef5025213b5bc44c5809"}
Jul 31 19:13:16.263074 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:16.263049    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:16.335324 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:16.335289    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca\") pod \"42c625ca-30c9-447f-9268-3aac2f870842\" (UID: \"42c625ca-30c9-447f-9268-3aac2f870842\") "
Jul 31 19:13:16.335677 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:16.335654    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42c625ca-30c9-447f-9268-3aac2f870842" (UID: "42c625ca-30c9-447f-9268-3aac2f870842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:13:16.436145 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:16.436057    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c625ca-30c9-447f-9268-3aac2f870842-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:17.143165 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:17.143131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj" event={"ID":"42c625ca-30c9-447f-9268-3aac2f870842","Type":"ContainerDied","Data":"ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8"}
Jul 31 19:13:17.143349 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:17.143174    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"
Jul 31 19:13:17.143349 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:17.143185    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff35216699088e693672678e2e28307e0ed22b048e8ee416c3998c392f923da8"
Jul 31 19:13:21.566723 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.566685    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:13:21.567405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.567377    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" podUID="b175d110-8feb-4a78-8c41-c38a7922f3bb" containerName="payload-pre-processing" containerID="cri-o://53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11" gracePeriod=30
Jul 31 19:13:21.727525 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.727488    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"]
Jul 31 19:13:21.732204 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.732169    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b30c0866-lrntj"]
Jul 31 19:13:21.817353 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.817298    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:13:21.881686 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.881659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume\") pod \"b175d110-8feb-4a78-8c41-c38a7922f3bb\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") "
Jul 31 19:13:21.881686 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.881692    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d75s7\" (UniqueName: \"kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7\") pod \"b175d110-8feb-4a78-8c41-c38a7922f3bb\" (UID: \"b175d110-8feb-4a78-8c41-c38a7922f3bb\") "
Jul 31 19:13:21.882073 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.882045    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b175d110-8feb-4a78-8c41-c38a7922f3bb" (UID: "b175d110-8feb-4a78-8c41-c38a7922f3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:13:21.886272 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.886236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7" (OuterVolumeSpecName: "kube-api-access-d75s7") pod "b175d110-8feb-4a78-8c41-c38a7922f3bb" (UID: "b175d110-8feb-4a78-8c41-c38a7922f3bb"). InnerVolumeSpecName "kube-api-access-d75s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:13:21.982488 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.982434    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b175d110-8feb-4a78-8c41-c38a7922f3bb-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:21.982488 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:21.982485    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75s7\" (UniqueName: \"kubernetes.io/projected/b175d110-8feb-4a78-8c41-c38a7922f3bb-kube-api-access-d75s7\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:22.169249 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.169214    2557 generic.go:356] "Generic (PLEG): container finished" podID="b175d110-8feb-4a78-8c41-c38a7922f3bb" containerID="53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11" exitCode=0
Jul 31 19:13:22.169422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.169257    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"
Jul 31 19:13:22.169422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.169303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" event={"ID":"b175d110-8feb-4a78-8c41-c38a7922f3bb","Type":"ContainerDied","Data":"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"}
Jul 31 19:13:22.169422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.169358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv" event={"ID":"b175d110-8feb-4a78-8c41-c38a7922f3bb","Type":"ContainerDied","Data":"3acd3d909cc9d76a6a16a3c9ddf5da104fe5900d3b851e4f468d471ea6dad3fa"}
Jul 31 19:13:22.169422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.169386    2557 scope.go:117] "RemoveContainer" containerID="53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"
Jul 31 19:13:22.178482 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.178432    2557 scope.go:117] "RemoveContainer" containerID="53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"
Jul 31 19:13:22.178983 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:13:22.178952    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11\": container with ID starting with 53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11 not found: ID does not exist" containerID="53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"
Jul 31 19:13:22.179078 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.178993    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11"} err="failed to get container status \"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11\": rpc error: code = NotFound desc = could not find container \"53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11\": container with ID starting with 53aba3ac6292f9ba980af443d355343991738cd73bc64331081b19de6e2a2a11 not found: ID does not exist"
Jul 31 19:13:22.182977 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.182948    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c625ca-30c9-447f-9268-3aac2f870842" path="/var/lib/kubelet/pods/42c625ca-30c9-447f-9268-3aac2f870842/volumes"
Jul 31 19:13:22.195675 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.195656    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:13:22.199957 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:22.199926    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b30c0866-769446bd9b-d4hnv"]
Jul 31 19:13:24.182642 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:24.182615    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b175d110-8feb-4a78-8c41-c38a7922f3bb" path="/var/lib/kubelet/pods/b175d110-8feb-4a78-8c41-c38a7922f3bb/volumes"
Jul 31 19:13:25.127625 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127596    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:25.127913 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127900    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b175d110-8feb-4a78-8c41-c38a7922f3bb" containerName="payload-pre-processing"
Jul 31 19:13:25.127960 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127915    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b175d110-8feb-4a78-8c41-c38a7922f3bb" containerName="payload-pre-processing"
Jul 31 19:13:25.127960 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127936    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c625ca-30c9-447f-9268-3aac2f870842" containerName="revoke-keys"
Jul 31 19:13:25.127960 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127941    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c625ca-30c9-447f-9268-3aac2f870842" containerName="revoke-keys"
Jul 31 19:13:25.128050 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.127995    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b175d110-8feb-4a78-8c41-c38a7922f3bb" containerName="payload-pre-processing"
Jul 31 19:13:25.128050 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.128003    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c625ca-30c9-447f-9268-3aac2f870842" containerName="revoke-keys"
Jul 31 19:13:25.131913 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.131895    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.134380 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.134361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b9249e59-serving-cert\""
Jul 31 19:13:25.134499 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.134476    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:13:25.134685 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.134669    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:13:25.135715 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.135699    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:13:25.140012 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.139592    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:25.215795 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.215758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.216202 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.215810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.216202 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.215903    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tpb\" (UniqueName: \"kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.316619 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.316576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.316821 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.316649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.316821 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.316701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tpb\" (UniqueName: \"kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.317315 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.317292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.319126 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.319108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.327027 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.327005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tpb\" (UniqueName: \"kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb\") pod \"maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.443397 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.443290    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:25.580164 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.580134    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:25.583345 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:13:25.583322    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5379829_3423_45ef_80da_48aa01683c13.slice/crio-54fdcf4170fdc8600d55b03f0ce4aecd0bb987c51f8ef65adc31d8a4de670e9a WatchSource:0}: Error finding container 54fdcf4170fdc8600d55b03f0ce4aecd0bb987c51f8ef65adc31d8a4de670e9a: Status 404 returned error can't find the container with id 54fdcf4170fdc8600d55b03f0ce4aecd0bb987c51f8ef65adc31d8a4de670e9a
Jul 31 19:13:25.808256 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.808175    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:25.812427 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.812408    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:25.815460 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.815423    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b9249e59-dockercfg-pdvcp\""
Jul 31 19:13:25.815551 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.815481    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b9249e59\""
Jul 31 19:13:25.824373 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.824345    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:25.920268 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.920236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:25.920459 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:25.920298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5lt\" (UniqueName: \"kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.021715 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.021676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5lt\" (UniqueName: \"kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.021898 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.021758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.022473 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.022423    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.034077 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.034054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5lt\" (UniqueName: \"kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt\") pod \"payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.122530 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.122495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:26.194399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.194365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" event={"ID":"a5379829-3423-45ef-80da-48aa01683c13","Type":"ContainerStarted","Data":"54fdcf4170fdc8600d55b03f0ce4aecd0bb987c51f8ef65adc31d8a4de670e9a"}
Jul 31 19:13:26.283210 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:26.283180    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:26.285337 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:13:26.285299    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56908c7a_1f8e_40ab_8479_e103e3f3a52e.slice/crio-c3998ba9f8d180126d81b4bdc57772baf722a99266091351f896c3bd09fc059d WatchSource:0}: Error finding container c3998ba9f8d180126d81b4bdc57772baf722a99266091351f896c3bd09fc059d: Status 404 returned error can't find the container with id c3998ba9f8d180126d81b4bdc57772baf722a99266091351f896c3bd09fc059d
Jul 31 19:13:27.203048 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:27.203011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" event={"ID":"56908c7a-1f8e-40ab-8479-e103e3f3a52e","Type":"ContainerStarted","Data":"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"}
Jul 31 19:13:27.203292 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:27.203079    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" event={"ID":"56908c7a-1f8e-40ab-8479-e103e3f3a52e","Type":"ContainerStarted","Data":"c3998ba9f8d180126d81b4bdc57772baf722a99266091351f896c3bd09fc059d"}
Jul 31 19:13:27.203292 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:27.203129    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:27.222337 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:27.222149    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" podStartSLOduration=2.222131798 podStartE2EDuration="2.222131798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:13:27.22072958 +0000 UTC m=+1785.568311461" watchObservedRunningTime="2026-07-31 19:13:27.222131798 +0000 UTC m=+1785.569713671"
Jul 31 19:13:28.209673 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:28.209637    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" event={"ID":"a5379829-3423-45ef-80da-48aa01683c13","Type":"ContainerStarted","Data":"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"}
Jul 31 19:13:28.210071 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:28.209930    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:28.229825 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:28.229773    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" podStartSLOduration=0.980120576 podStartE2EDuration="3.229758508s" totalImagesPullingTime="2.249637932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:13:28.2276615 +0000 UTC m=+1786.575243406" watchObservedRunningTime="2026-07-31 19:13:28.229758508 +0000 UTC m=+1786.577340421"
Jul 31 19:13:34.220558 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:34.220489    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:38.212349 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:38.212314    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:42.236576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:42.236542    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:13:42.238432 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:42.238407    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:13:49.999305 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:49.999266    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"]
Jul 31 19:13:50.007502 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.007477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:50.010382 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.010353    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:13:50.011205 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.011179    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"]
Jul 31 19:13:50.031486 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.031424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9249e59-r557b\" (UID: \"1ed22702-e32a-4c64-8d5d-2a85cb348aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:50.132708 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.132665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9249e59-r557b\" (UID: \"1ed22702-e32a-4c64-8d5d-2a85cb348aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:50.133243 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.133224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9249e59-r557b\" (UID: \"1ed22702-e32a-4c64-8d5d-2a85cb348aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:50.319400 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.319314    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:50.451313 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:50.451284    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"]
Jul 31 19:13:50.453115 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:13:50.453089    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed22702_e32a_4c64_8d5d_2a85cb348aeb.slice/crio-4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63 WatchSource:0}: Error finding container 4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63: Status 404 returned error can't find the container with id 4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63
Jul 31 19:13:51.323682 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:51.323650    2557 generic.go:356] "Generic (PLEG): container finished" podID="1ed22702-e32a-4c64-8d5d-2a85cb348aeb" containerID="4d1258a0918194371c72fd82877fe44750e7871190e8723c354217049064830d" exitCode=0
Jul 31 19:13:51.324055 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:51.323708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b" event={"ID":"1ed22702-e32a-4c64-8d5d-2a85cb348aeb","Type":"ContainerDied","Data":"4d1258a0918194371c72fd82877fe44750e7871190e8723c354217049064830d"}
Jul 31 19:13:51.324055 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:51.323757    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b" event={"ID":"1ed22702-e32a-4c64-8d5d-2a85cb348aeb","Type":"ContainerStarted","Data":"4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63"}
Jul 31 19:13:52.461339 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:52.461310    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:52.553320 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:52.553283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca\") pod \"1ed22702-e32a-4c64-8d5d-2a85cb348aeb\" (UID: \"1ed22702-e32a-4c64-8d5d-2a85cb348aeb\") "
Jul 31 19:13:52.553731 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:52.553704    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ed22702-e32a-4c64-8d5d-2a85cb348aeb" (UID: "1ed22702-e32a-4c64-8d5d-2a85cb348aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:13:52.653888 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:52.653853    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed22702-e32a-4c64-8d5d-2a85cb348aeb-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:53.333489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:53.333436    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"
Jul 31 19:13:53.333489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:53.333480    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b" event={"ID":"1ed22702-e32a-4c64-8d5d-2a85cb348aeb","Type":"ContainerDied","Data":"4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63"}
Jul 31 19:13:53.333708 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:53.333536    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab70aa58de8e690533fdfa502aef967c0a434913685ce6b1cee0396c5edce63"
Jul 31 19:13:56.652139 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.652099    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:56.652663 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.652412    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" podUID="a5379829-3423-45ef-80da-48aa01683c13" containerName="maas-api" containerID="cri-o://eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba" gracePeriod=30
Jul 31 19:13:56.714728 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.714694    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:56.714953 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.714910    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" podUID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" containerName="payload-processing" containerID="cri-o://97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8" gracePeriod=30
Jul 31 19:13:56.894120 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.894061    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"]
Jul 31 19:13:56.897236 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.897205    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9249e59-r557b"]
Jul 31 19:13:56.972068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.972047    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:56.983394 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:56.983370    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:57.095318 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095282    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls\") pod \"a5379829-3423-45ef-80da-48aa01683c13\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") "
Jul 31 19:13:57.095318 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume\") pod \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") "
Jul 31 19:13:57.095791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095377    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7tpb\" (UniqueName: \"kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb\") pod \"a5379829-3423-45ef-80da-48aa01683c13\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") "
Jul 31 19:13:57.095791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095529    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca\") pod \"a5379829-3423-45ef-80da-48aa01683c13\" (UID: \"a5379829-3423-45ef-80da-48aa01683c13\") "
Jul 31 19:13:57.095791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095586    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq5lt\" (UniqueName: \"kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt\") pod \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\" (UID: \"56908c7a-1f8e-40ab-8479-e103e3f3a52e\") "
Jul 31 19:13:57.096208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095795    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56908c7a-1f8e-40ab-8479-e103e3f3a52e" (UID: "56908c7a-1f8e-40ab-8479-e103e3f3a52e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:13:57.096208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095803    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5379829-3423-45ef-80da-48aa01683c13" (UID: "a5379829-3423-45ef-80da-48aa01683c13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:13:57.096208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095886    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5379829-3423-45ef-80da-48aa01683c13-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:57.096208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.095904    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56908c7a-1f8e-40ab-8479-e103e3f3a52e-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:57.099649 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.099616    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt" (OuterVolumeSpecName: "kube-api-access-wq5lt") pod "56908c7a-1f8e-40ab-8479-e103e3f3a52e" (UID: "56908c7a-1f8e-40ab-8479-e103e3f3a52e"). InnerVolumeSpecName "kube-api-access-wq5lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:13:57.099649 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.099637    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb" (OuterVolumeSpecName: "kube-api-access-k7tpb") pod "a5379829-3423-45ef-80da-48aa01683c13" (UID: "a5379829-3423-45ef-80da-48aa01683c13"). InnerVolumeSpecName "kube-api-access-k7tpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:13:57.099783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.099654    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5379829-3423-45ef-80da-48aa01683c13" (UID: "a5379829-3423-45ef-80da-48aa01683c13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:13:57.196617 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.196536    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7tpb\" (UniqueName: \"kubernetes.io/projected/a5379829-3423-45ef-80da-48aa01683c13-kube-api-access-k7tpb\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:57.196617 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.196566    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5lt\" (UniqueName: \"kubernetes.io/projected/56908c7a-1f8e-40ab-8479-e103e3f3a52e-kube-api-access-wq5lt\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:57.196617 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.196581    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5379829-3423-45ef-80da-48aa01683c13-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:13:57.352733 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.352695    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5379829-3423-45ef-80da-48aa01683c13" containerID="eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba" exitCode=0
Jul 31 19:13:57.352733 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.352728    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"
Jul 31 19:13:57.352955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.352774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" event={"ID":"a5379829-3423-45ef-80da-48aa01683c13","Type":"ContainerDied","Data":"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"}
Jul 31 19:13:57.352955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.352830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2" event={"ID":"a5379829-3423-45ef-80da-48aa01683c13","Type":"ContainerDied","Data":"54fdcf4170fdc8600d55b03f0ce4aecd0bb987c51f8ef65adc31d8a4de670e9a"}
Jul 31 19:13:57.352955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.352857    2557 scope.go:117] "RemoveContainer" containerID="eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"
Jul 31 19:13:57.354254 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.354232    2557 generic.go:356] "Generic (PLEG): container finished" podID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" containerID="97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8" exitCode=0
Jul 31 19:13:57.354381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.354272    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"
Jul 31 19:13:57.354381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.354320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" event={"ID":"56908c7a-1f8e-40ab-8479-e103e3f3a52e","Type":"ContainerDied","Data":"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"}
Jul 31 19:13:57.354381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.354357    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp" event={"ID":"56908c7a-1f8e-40ab-8479-e103e3f3a52e","Type":"ContainerDied","Data":"c3998ba9f8d180126d81b4bdc57772baf722a99266091351f896c3bd09fc059d"}
Jul 31 19:13:57.362083 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.361945    2557 scope.go:117] "RemoveContainer" containerID="eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"
Jul 31 19:13:57.362343 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:13:57.362322    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba\": container with ID starting with eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba not found: ID does not exist" containerID="eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"
Jul 31 19:13:57.362412 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.362352    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba"} err="failed to get container status \"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba\": rpc error: code = NotFound desc = could not find container \"eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba\": container with ID starting with eed1c8461a74753bda9b55d7afe06ace78fc9bca62120b6dc630dec2921531ba not found: ID does not exist"
Jul 31 19:13:57.362412 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.362374    2557 scope.go:117] "RemoveContainer" containerID="97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"
Jul 31 19:13:57.370746 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.370731    2557 scope.go:117] "RemoveContainer" containerID="97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"
Jul 31 19:13:57.371012 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:13:57.370996    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8\": container with ID starting with 97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8 not found: ID does not exist" containerID="97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"
Jul 31 19:13:57.371058 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.371019    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8"} err="failed to get container status \"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8\": rpc error: code = NotFound desc = could not find container \"97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8\": container with ID starting with 97d7d9262b0b89a13869cd962d735df630715de0367a0d008c0b7682ed96aca8 not found: ID does not exist"
Jul 31 19:13:57.381140 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.381120    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:57.386291 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.386271    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b9249e59-667f4c6f78-xxcr2"]
Jul 31 19:13:57.400300 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.400272    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:57.407650 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:57.407621    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9249e59-b876d8569-m7gmp"]
Jul 31 19:13:58.184016 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:58.183982    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed22702-e32a-4c64-8d5d-2a85cb348aeb" path="/var/lib/kubelet/pods/1ed22702-e32a-4c64-8d5d-2a85cb348aeb/volumes"
Jul 31 19:13:58.184403 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:58.184333    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" path="/var/lib/kubelet/pods/56908c7a-1f8e-40ab-8479-e103e3f3a52e/volumes"
Jul 31 19:13:58.184692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:13:58.184678    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5379829-3423-45ef-80da-48aa01683c13" path="/var/lib/kubelet/pods/a5379829-3423-45ef-80da-48aa01683c13/volumes"
Jul 31 19:14:01.251165 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251132    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251465    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5379829-3423-45ef-80da-48aa01683c13" containerName="maas-api"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251477    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5379829-3423-45ef-80da-48aa01683c13" containerName="maas-api"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251487    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed22702-e32a-4c64-8d5d-2a85cb348aeb" containerName="revoke-keys"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251492    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed22702-e32a-4c64-8d5d-2a85cb348aeb" containerName="revoke-keys"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251506    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" containerName="payload-processing"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251512    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" containerName="payload-processing"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251575    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="56908c7a-1f8e-40ab-8479-e103e3f3a52e" containerName="payload-processing"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251582    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5379829-3423-45ef-80da-48aa01683c13" containerName="maas-api"
Jul 31 19:14:01.251626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.251588    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed22702-e32a-4c64-8d5d-2a85cb348aeb" containerName="revoke-keys"
Jul 31 19:14:01.258005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.257984    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.260721 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.260689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f3556860-serving-cert\""
Jul 31 19:14:01.260846 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.260826    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:14:01.262062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.262035    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:14:01.262190 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.262035    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:14:01.264638 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.264612    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:01.333353 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.333303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.333564 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.333393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.333564 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.333506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.434313 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.434281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.434313 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.434326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.434599 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.434367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.434599 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:01.434502    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-serving-cert: secret "maas-api-e2e-ait-f3556860-serving-cert" not found
Jul 31 19:14:01.434599 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:01.434576    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls podName:3148e329-b343-43b2-bea8-6413125f5b8e nodeName:}" failed. No retries permitted until 2026-07-31 19:14:01.93455885 +0000 UTC m=+1820.282140708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls") pod "maas-api-e2e-ait-f3556860-5985987d79-bmfn6" (UID: "3148e329-b343-43b2-bea8-6413125f5b8e") : secret "maas-api-e2e-ait-f3556860-serving-cert" not found
Jul 31 19:14:01.435059 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.435039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.444197 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.444171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.896665 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.896614    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:01.901104 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.901085    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:01.904301 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.904273    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f3556860\""
Jul 31 19:14:01.910089 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.910060    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:01.919917 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.919886    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:01.923646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.923622    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:01.926666 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.926646    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f3556860-dockercfg-4qdxz\""
Jul 31 19:14:01.936252 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.936227    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:01.939716 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:01.939693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:01.939855 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:01.939837    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-serving-cert: secret "maas-api-e2e-ait-f3556860-serving-cert" not found
Jul 31 19:14:01.939910 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:01.939897    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls podName:3148e329-b343-43b2-bea8-6413125f5b8e nodeName:}" failed. No retries permitted until 2026-07-31 19:14:02.939881356 +0000 UTC m=+1821.287463214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls") pod "maas-api-e2e-ait-f3556860-5985987d79-bmfn6" (UID: "3148e329-b343-43b2-bea8-6413125f5b8e") : secret "maas-api-e2e-ait-f3556860-serving-cert" not found
Jul 31 19:14:02.040422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.040392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.040606 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.040465    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4pp\" (UniqueName: \"kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.040606 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.040492    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmv5\" (UniqueName: \"kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.040606 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.040546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.140994 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.140956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.141171 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.141010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4pp\" (UniqueName: \"kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.141171 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.141128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmv5\" (UniqueName: \"kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.141303 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.141234    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.141750 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.141733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.141944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.141928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.150520 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.150429    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4pp\" (UniqueName: \"kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp\") pod \"payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.150646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.150577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmv5\" (UniqueName: \"kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5\") pod \"payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") " pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.213670 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.213628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:02.235554 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.235493    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:02.377410 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.377215    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:02.379726 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:14:02.379683    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c3ef96_c0bc_4f2a_949a_6cc2ad5e4027.slice/crio-cba55cf90be4f88f357f22b2d1c6efba6c9cdce146bd5361b4bca22aac90ccff WatchSource:0}: Error finding container cba55cf90be4f88f357f22b2d1c6efba6c9cdce146bd5361b4bca22aac90ccff: Status 404 returned error can't find the container with id cba55cf90be4f88f357f22b2d1c6efba6c9cdce146bd5361b4bca22aac90ccff
Jul 31 19:14:02.408084 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.408060    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:02.410642 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:14:02.410612    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfca0cd8_d404_4b30_a497_bb1e8d0dd141.slice/crio-f7ef7142fff8e7ad882577175d8ef313681d61f018a62f59223779775da32263 WatchSource:0}: Error finding container f7ef7142fff8e7ad882577175d8ef313681d61f018a62f59223779775da32263: Status 404 returned error can't find the container with id f7ef7142fff8e7ad882577175d8ef313681d61f018a62f59223779775da32263
Jul 31 19:14:02.949405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.949368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:02.952010 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:02.951983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"maas-api-e2e-ait-f3556860-5985987d79-bmfn6\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:03.070741 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.070696    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:03.208238 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.208213    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:03.210591 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:14:03.210563    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3148e329_b343_43b2_bea8_6413125f5b8e.slice/crio-f3b360a1d08a8ebed4d75b9d9615897c46c819cbcfd837025db72b6bc0b7257a WatchSource:0}: Error finding container f3b360a1d08a8ebed4d75b9d9615897c46c819cbcfd837025db72b6bc0b7257a: Status 404 returned error can't find the container with id f3b360a1d08a8ebed4d75b9d9615897c46c819cbcfd837025db72b6bc0b7257a
Jul 31 19:14:03.391294 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.391258    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" event={"ID":"bfca0cd8-d404-4b30-a497-bb1e8d0dd141","Type":"ContainerStarted","Data":"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"}
Jul 31 19:14:03.391775 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.391319    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" event={"ID":"bfca0cd8-d404-4b30-a497-bb1e8d0dd141","Type":"ContainerStarted","Data":"f7ef7142fff8e7ad882577175d8ef313681d61f018a62f59223779775da32263"}
Jul 31 19:14:03.391775 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.391428    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:03.392906 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.392884    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" event={"ID":"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027","Type":"ContainerStarted","Data":"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"}
Jul 31 19:14:03.393023 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.392927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" event={"ID":"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027","Type":"ContainerStarted","Data":"cba55cf90be4f88f357f22b2d1c6efba6c9cdce146bd5361b4bca22aac90ccff"}
Jul 31 19:14:03.393023 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.392955    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:03.394035 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.394014    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" event={"ID":"3148e329-b343-43b2-bea8-6413125f5b8e","Type":"ContainerStarted","Data":"f3b360a1d08a8ebed4d75b9d9615897c46c819cbcfd837025db72b6bc0b7257a"}
Jul 31 19:14:03.412570 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.412519    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" podStartSLOduration=2.412503041 podStartE2EDuration="2.412503041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:03.41188594 +0000 UTC m=+1821.759467823" watchObservedRunningTime="2026-07-31 19:14:03.412503041 +0000 UTC m=+1821.760084921"
Jul 31 19:14:03.432279 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:03.432227    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" podStartSLOduration=2.432211254 podStartE2EDuration="2.432211254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:03.430419784 +0000 UTC m=+1821.778001664" watchObservedRunningTime="2026-07-31 19:14:03.432211254 +0000 UTC m=+1821.779793134"
Jul 31 19:14:04.400846 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:04.400806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" event={"ID":"3148e329-b343-43b2-bea8-6413125f5b8e","Type":"ContainerStarted","Data":"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"}
Jul 31 19:14:04.401274 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:04.400992    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:04.421464 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:04.421388    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" podStartSLOduration=2.806861561 podStartE2EDuration="3.421372088s" totalImagesPullingTime="614.510527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:04.419207356 +0000 UTC m=+1822.766789241" watchObservedRunningTime="2026-07-31 19:14:04.421372088 +0000 UTC m=+1822.768953968"
Jul 31 19:14:10.411559 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:10.411530    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:14.403421 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:14.403390    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:14.403875 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:14.403488    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:41.993271 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:41.993234    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:41.993887 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:41.993541    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" podUID="3148e329-b343-43b2-bea8-6413125f5b8e" containerName="maas-api" containerID="cri-o://34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad" gracePeriod=30
Jul 31 19:14:42.052816 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.052778    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:42.053107 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.053061    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" podUID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" containerName="payload-processing" containerID="cri-o://d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff" gracePeriod=30
Jul 31 19:14:42.060641 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.060610    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:42.060899 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.060868    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" podUID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" containerName="payload-pre-processing" containerID="cri-o://e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0" gracePeriod=30
Jul 31 19:14:42.314193 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.314161    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:42.336262 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.335993    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:42.346701 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.346672    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:42.383106 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca\") pod \"3148e329-b343-43b2-bea8-6413125f5b8e\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") "
Jul 31 19:14:42.383106 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383112    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") pod \"3148e329-b343-43b2-bea8-6413125f5b8e\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") "
Jul 31 19:14:42.383330 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383133    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr4pp\" (UniqueName: \"kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp\") pod \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") "
Jul 31 19:14:42.383330 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383163    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume\") pod \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") "
Jul 31 19:14:42.383467 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383339    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsmv5\" (UniqueName: \"kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5\") pod \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\" (UID: \"bfca0cd8-d404-4b30-a497-bb1e8d0dd141\") "
Jul 31 19:14:42.383467 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383386    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9\") pod \"3148e329-b343-43b2-bea8-6413125f5b8e\" (UID: \"3148e329-b343-43b2-bea8-6413125f5b8e\") "
Jul 31 19:14:42.383580 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383530    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume\") pod \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\" (UID: \"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027\") "
Jul 31 19:14:42.383580 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383566    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfca0cd8-d404-4b30-a497-bb1e8d0dd141" (UID: "bfca0cd8-d404-4b30-a497-bb1e8d0dd141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:14:42.383684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3148e329-b343-43b2-bea8-6413125f5b8e" (UID: "3148e329-b343-43b2-bea8-6413125f5b8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:14:42.383856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383836    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3148e329-b343-43b2-bea8-6413125f5b8e-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.383916 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.383861    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.384147 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.384122    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" (UID: "f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:14:42.385522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.385496    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp" (OuterVolumeSpecName: "kube-api-access-jr4pp") pod "f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" (UID: "f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027"). InnerVolumeSpecName "kube-api-access-jr4pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:14:42.385750 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.385728    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9" (OuterVolumeSpecName: "kube-api-access-9sfb9") pod "3148e329-b343-43b2-bea8-6413125f5b8e" (UID: "3148e329-b343-43b2-bea8-6413125f5b8e"). InnerVolumeSpecName "kube-api-access-9sfb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:14:42.386019 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.386004    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3148e329-b343-43b2-bea8-6413125f5b8e" (UID: "3148e329-b343-43b2-bea8-6413125f5b8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:14:42.386094 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.386079    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5" (OuterVolumeSpecName: "kube-api-access-hsmv5") pod "bfca0cd8-d404-4b30-a497-bb1e8d0dd141" (UID: "bfca0cd8-d404-4b30-a497-bb1e8d0dd141"). InnerVolumeSpecName "kube-api-access-hsmv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:14:42.484841 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.484796    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4pp\" (UniqueName: \"kubernetes.io/projected/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-kube-api-access-jr4pp\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.484841 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.484832    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmv5\" (UniqueName: \"kubernetes.io/projected/bfca0cd8-d404-4b30-a497-bb1e8d0dd141-kube-api-access-hsmv5\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.484841 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.484844    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfb9\" (UniqueName: \"kubernetes.io/projected/3148e329-b343-43b2-bea8-6413125f5b8e-kube-api-access-9sfb9\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.485124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.484856    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.485124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.484869    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3148e329-b343-43b2-bea8-6413125f5b8e-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:14:42.582865 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.582785    2557 generic.go:356] "Generic (PLEG): container finished" podID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" containerID="d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff" exitCode=0
Jul 31 19:14:42.582865 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.582837    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"
Jul 31 19:14:42.583076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.582866    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" event={"ID":"bfca0cd8-d404-4b30-a497-bb1e8d0dd141","Type":"ContainerDied","Data":"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"}
Jul 31 19:14:42.583076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.582915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt" event={"ID":"bfca0cd8-d404-4b30-a497-bb1e8d0dd141","Type":"ContainerDied","Data":"f7ef7142fff8e7ad882577175d8ef313681d61f018a62f59223779775da32263"}
Jul 31 19:14:42.583076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.582943    2557 scope.go:117] "RemoveContainer" containerID="d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"
Jul 31 19:14:42.584229 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.584200    2557 generic.go:356] "Generic (PLEG): container finished" podID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" containerID="e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0" exitCode=0
Jul 31 19:14:42.584229 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.584222    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" event={"ID":"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027","Type":"ContainerDied","Data":"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"}
Jul 31 19:14:42.584390 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.584255    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"
Jul 31 19:14:42.584390 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.584280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv" event={"ID":"f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027","Type":"ContainerDied","Data":"cba55cf90be4f88f357f22b2d1c6efba6c9cdce146bd5361b4bca22aac90ccff"}
Jul 31 19:14:42.585649 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.585631    2557 generic.go:356] "Generic (PLEG): container finished" podID="3148e329-b343-43b2-bea8-6413125f5b8e" containerID="34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad" exitCode=0
Jul 31 19:14:42.585769 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.585673    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"
Jul 31 19:14:42.585769 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.585686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" event={"ID":"3148e329-b343-43b2-bea8-6413125f5b8e","Type":"ContainerDied","Data":"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"}
Jul 31 19:14:42.585769 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.585714    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6" event={"ID":"3148e329-b343-43b2-bea8-6413125f5b8e","Type":"ContainerDied","Data":"f3b360a1d08a8ebed4d75b9d9615897c46c819cbcfd837025db72b6bc0b7257a"}
Jul 31 19:14:42.593587 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.593552    2557 scope.go:117] "RemoveContainer" containerID="d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"
Jul 31 19:14:42.593888 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:42.593864    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff\": container with ID starting with d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff not found: ID does not exist" containerID="d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"
Jul 31 19:14:42.593972 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.593895    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff"} err="failed to get container status \"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff\": rpc error: code = NotFound desc = could not find container \"d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff\": container with ID starting with d81884ac29a6213ed074b3b2ac7261d33602394418df903248590e89650f8fff not found: ID does not exist"
Jul 31 19:14:42.593972 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.593918    2557 scope.go:117] "RemoveContainer" containerID="e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"
Jul 31 19:14:42.602264 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.602247    2557 scope.go:117] "RemoveContainer" containerID="e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"
Jul 31 19:14:42.602543 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:42.602522    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0\": container with ID starting with e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0 not found: ID does not exist" containerID="e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"
Jul 31 19:14:42.602584 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.602556    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0"} err="failed to get container status \"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0\": rpc error: code = NotFound desc = could not find container \"e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0\": container with ID starting with e24e1dadb17c1f00c76cf32373462e7770bac4d8d43fdd753a71ed33022261c0 not found: ID does not exist"
Jul 31 19:14:42.602584 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.602579    2557 scope.go:117] "RemoveContainer" containerID="34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"
Jul 31 19:14:42.611387 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.611349    2557 scope.go:117] "RemoveContainer" containerID="34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"
Jul 31 19:14:42.611924 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:42.611895    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad\": container with ID starting with 34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad not found: ID does not exist" containerID="34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"
Jul 31 19:14:42.612023 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.611934    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad"} err="failed to get container status \"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad\": rpc error: code = NotFound desc = could not find container \"34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad\": container with ID starting with 34a052f7826bef368dcd96a3017e343e6b7c74aedfae6684c25ba81038c844ad not found: ID does not exist"
Jul 31 19:14:42.614315 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.614295    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:42.623891 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.623863    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3556860-84d67d5cbb-6wvtt"]
Jul 31 19:14:42.634134 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.634109    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:42.640658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.640639    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3556860-6d474bfc68-mrckv"]
Jul 31 19:14:42.653852 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.653821    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:42.659053 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:42.659029    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3556860-5985987d79-bmfn6"]
Jul 31 19:14:44.184220 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:44.184183    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3148e329-b343-43b2-bea8-6413125f5b8e" path="/var/lib/kubelet/pods/3148e329-b343-43b2-bea8-6413125f5b8e/volumes"
Jul 31 19:14:44.184771 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:44.184657    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" path="/var/lib/kubelet/pods/bfca0cd8-d404-4b30-a497-bb1e8d0dd141/volumes"
Jul 31 19:14:44.185069 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:44.185054    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" path="/var/lib/kubelet/pods/f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027/volumes"
Jul 31 19:14:47.520180 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520146    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520465    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" containerName="payload-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520477    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" containerName="payload-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520487    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3148e329-b343-43b2-bea8-6413125f5b8e" containerName="maas-api"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520493    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3148e329-b343-43b2-bea8-6413125f5b8e" containerName="maas-api"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520502    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" containerName="payload-pre-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520511    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" containerName="payload-pre-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520571    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c3ef96-c0bc-4f2a-949a-6cc2ad5e4027" containerName="payload-pre-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520578    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfca0cd8-d404-4b30-a497-bb1e8d0dd141" containerName="payload-processing"
Jul 31 19:14:47.520688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.520584    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3148e329-b343-43b2-bea8-6413125f5b8e" containerName="maas-api"
Jul 31 19:14:47.524851 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.524828    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.527611 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.527591    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:14:47.527733 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.527593    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:14:47.528592 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.528575    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:14:47.528592 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.528586    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bc42f1be-serving-cert\""
Jul 31 19:14:47.534291 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.533432    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:14:47.626544 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.626497    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.626791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.626678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.626791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.626706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclkq\" (UniqueName: \"kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.727463 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.727406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.727463 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.727469    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclkq\" (UniqueName: \"kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.727681 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.727531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.727681 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:47.727647    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-serving-cert: secret "maas-api-e2e-derive-bc42f1be-serving-cert" not found
Jul 31 19:14:47.727747 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:14:47.727716    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls podName:ea3aea66-7e6e-45d1-86e7-07c08d34122d nodeName:}" failed. No retries permitted until 2026-07-31 19:14:48.227694797 +0000 UTC m=+1866.575276658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls") pod "maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" (UID: "ea3aea66-7e6e-45d1-86e7-07c08d34122d") : secret "maas-api-e2e-derive-bc42f1be-serving-cert" not found
Jul 31 19:14:47.728050 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.728030    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:47.737320 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:47.737293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclkq\" (UniqueName: \"kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:48.233320 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.233282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:48.235897 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.235865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") pod \"maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:48.309891 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.309855    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:14:48.313429 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.313405    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.316297 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.316276    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bc42f1be-dockercfg-8zb5b\""
Jul 31 19:14:48.316414 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.316279    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bc42f1be\""
Jul 31 19:14:48.325116 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.325092    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:14:48.327293 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.327267    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:14:48.330460 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.330429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.340576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.340487    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:14:48.435288 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.435247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.435522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.435324    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw85\" (UniqueName: \"kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.435522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.435381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgg9p\" (UniqueName: \"kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.435522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.435419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.438184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.438159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:48.536015 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.535983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw85\" (UniqueName: \"kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.536458 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.536029    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgg9p\" (UniqueName: \"kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.536458 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.536060    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.536458 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.536247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.537076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.537038    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.537190 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.537105    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.547113 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.547076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgg9p\" (UniqueName: \"kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p\") pod \"payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") " pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.547259 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.547169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw85\" (UniqueName: \"kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85\") pod \"payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.579231 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.579078    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:14:48.618896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.618859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" event={"ID":"ea3aea66-7e6e-45d1-86e7-07c08d34122d","Type":"ContainerStarted","Data":"fa3a4a89e0002d21d10f48193c114e73d22da987cddee80805fa6b689bfb3dc2"}
Jul 31 19:14:48.625000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.624963    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:48.642490 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.642461    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:48.771884 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.771728    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:14:48.774971 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:14:48.774944    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496ed0d_093d_4a3b_9cf0_0c4f438432da.slice/crio-3ba79993d947b0ae8f9d0652d5d85bf1a05d40fe43789371fe558d762a8fb661 WatchSource:0}: Error finding container 3ba79993d947b0ae8f9d0652d5d85bf1a05d40fe43789371fe558d762a8fb661: Status 404 returned error can't find the container with id 3ba79993d947b0ae8f9d0652d5d85bf1a05d40fe43789371fe558d762a8fb661
Jul 31 19:14:48.799310 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:48.799284    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:14:48.808677 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:14:48.808644    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d544891_3b46_42f1_ac02_491239ff1d4f.slice/crio-318791d72d234a765b2028a2e15333d0c9ecd071620e55fc2ebe421736b909a6 WatchSource:0}: Error finding container 318791d72d234a765b2028a2e15333d0c9ecd071620e55fc2ebe421736b909a6: Status 404 returned error can't find the container with id 318791d72d234a765b2028a2e15333d0c9ecd071620e55fc2ebe421736b909a6
Jul 31 19:14:49.626636 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.626587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" event={"ID":"9d544891-3b46-42f1-ac02-491239ff1d4f","Type":"ContainerStarted","Data":"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"}
Jul 31 19:14:49.627114 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.626647    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" event={"ID":"9d544891-3b46-42f1-ac02-491239ff1d4f","Type":"ContainerStarted","Data":"318791d72d234a765b2028a2e15333d0c9ecd071620e55fc2ebe421736b909a6"}
Jul 31 19:14:49.627114 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.626675    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:14:49.628138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.628116    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" event={"ID":"ea3aea66-7e6e-45d1-86e7-07c08d34122d","Type":"ContainerStarted","Data":"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"}
Jul 31 19:14:49.628247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.628198    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:14:49.629713 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.629689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" event={"ID":"3496ed0d-093d-4a3b-9cf0-0c4f438432da","Type":"ContainerStarted","Data":"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"}
Jul 31 19:14:49.629801 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.629730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" event={"ID":"3496ed0d-093d-4a3b-9cf0-0c4f438432da","Type":"ContainerStarted","Data":"3ba79993d947b0ae8f9d0652d5d85bf1a05d40fe43789371fe558d762a8fb661"}
Jul 31 19:14:49.629801 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.629759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:14:49.645605 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.645560    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" podStartSLOduration=1.645545767 podStartE2EDuration="1.645545767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:49.643961947 +0000 UTC m=+1867.991543828" watchObservedRunningTime="2026-07-31 19:14:49.645545767 +0000 UTC m=+1867.993127647"
Jul 31 19:14:49.662348 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.662296    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" podStartSLOduration=1.662279781 podStartE2EDuration="1.662279781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:49.660309636 +0000 UTC m=+1868.007891529" watchObservedRunningTime="2026-07-31 19:14:49.662279781 +0000 UTC m=+1868.009861660"
Jul 31 19:14:49.680231 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:49.680171    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" podStartSLOduration=2.324675355 podStartE2EDuration="2.680149767s" totalImagesPullingTime="355.474412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:14:49.678524341 +0000 UTC m=+1868.026106222" watchObservedRunningTime="2026-07-31 19:14:49.680149767 +0000 UTC m=+1868.027731655"
Jul 31 19:14:55.639713 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:14:55.639685    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:15:00.148676 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.148641    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"]
Jul 31 19:15:00.153558 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.153534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"
Jul 31 19:15:00.156521 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.156488    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:15:00.165334 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.165311    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"
Jul 31 19:15:00.187991 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.187965    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"]
Jul 31 19:15:00.311882 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.311852    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"]
Jul 31 19:15:00.313050 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:15:00.313021    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c61f32b_9cc3_459e_98e2_f125be421cb7.slice/crio-5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00 WatchSource:0}: Error finding container 5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00: Status 404 returned error can't find the container with id 5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00
Jul 31 19:15:00.636514 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.636484    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:15:00.636678 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.636553    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:15:00.682176 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.682136    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g" event={"ID":"0c61f32b-9cc3-459e-98e2-f125be421cb7","Type":"ContainerStarted","Data":"b50fda1c5bb202d56491da1045d41daf544da6869394594d3f3d8e771d56a6d8"}
Jul 31 19:15:00.682365 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.682203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g" event={"ID":"0c61f32b-9cc3-459e-98e2-f125be421cb7","Type":"ContainerStarted","Data":"5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00"}
Jul 31 19:15:00.707552 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:00.707501    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g" podStartSLOduration=0.707484251 podStartE2EDuration="707.484251ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:15:00.706348852 +0000 UTC m=+1879.053930732" watchObservedRunningTime="2026-07-31 19:15:00.707484251 +0000 UTC m=+1879.055066132"
Jul 31 19:15:01.688632 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:01.688604    2557 generic.go:356] "Generic (PLEG): container finished" podID="0c61f32b-9cc3-459e-98e2-f125be421cb7" containerID="b50fda1c5bb202d56491da1045d41daf544da6869394594d3f3d8e771d56a6d8" exitCode=0
Jul 31 19:15:01.688997 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:01.688695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g" event={"ID":"0c61f32b-9cc3-459e-98e2-f125be421cb7","Type":"ContainerDied","Data":"b50fda1c5bb202d56491da1045d41daf544da6869394594d3f3d8e771d56a6d8"}
Jul 31 19:15:02.820337 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:02.820312    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"
Jul 31 19:15:03.699261 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:03.699182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g" event={"ID":"0c61f32b-9cc3-459e-98e2-f125be421cb7","Type":"ContainerDied","Data":"5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00"}
Jul 31 19:15:03.699261 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:03.699209    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758755-xfs6g"
Jul 31 19:15:03.699261 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:03.699228    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3a13b80573b17267f02285d2b109f20d94343dc7dea075e6527968986c6a00"
Jul 31 19:15:11.293980 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.293943    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"]
Jul 31 19:15:11.294381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.294255    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c61f32b-9cc3-459e-98e2-f125be421cb7" containerName="cleanup"
Jul 31 19:15:11.294381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.294266    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c61f32b-9cc3-459e-98e2-f125be421cb7" containerName="cleanup"
Jul 31 19:15:11.294381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.294318    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c61f32b-9cc3-459e-98e2-f125be421cb7" containerName="cleanup"
Jul 31 19:15:11.297300 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.297283    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:11.300208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.300190    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:15:11.306162 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.306134    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"]
Jul 31 19:15:11.438346 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.438309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk\" (UID: \"8816a3e6-2fb9-4137-8db1-1fda30486a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:11.539004 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.538959    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk\" (UID: \"8816a3e6-2fb9-4137-8db1-1fda30486a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:11.539594 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.539575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk\" (UID: \"8816a3e6-2fb9-4137-8db1-1fda30486a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:11.608306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.608279    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:11.742311 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:11.742287    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"]
Jul 31 19:15:11.744179 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:15:11.744150    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8816a3e6_2fb9_4137_8db1_1fda30486a11.slice/crio-e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb WatchSource:0}: Error finding container e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb: Status 404 returned error can't find the container with id e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb
Jul 31 19:15:12.744149 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:12.744055    2557 generic.go:356] "Generic (PLEG): container finished" podID="8816a3e6-2fb9-4137-8db1-1fda30486a11" containerID="6c45677ef4a0ca39205166b0c08acdbec988bd70a42b1de637a240964d9c0381" exitCode=0
Jul 31 19:15:12.744527 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:12.744141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk" event={"ID":"8816a3e6-2fb9-4137-8db1-1fda30486a11","Type":"ContainerDied","Data":"6c45677ef4a0ca39205166b0c08acdbec988bd70a42b1de637a240964d9c0381"}
Jul 31 19:15:12.744527 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:12.744197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk" event={"ID":"8816a3e6-2fb9-4137-8db1-1fda30486a11","Type":"ContainerStarted","Data":"e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb"}
Jul 31 19:15:13.880326 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:13.880298    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:13.960536 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:13.960507    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca\") pod \"8816a3e6-2fb9-4137-8db1-1fda30486a11\" (UID: \"8816a3e6-2fb9-4137-8db1-1fda30486a11\") "
Jul 31 19:15:13.960869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:13.960844    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8816a3e6-2fb9-4137-8db1-1fda30486a11" (UID: "8816a3e6-2fb9-4137-8db1-1fda30486a11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:15:14.061104 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:14.061022    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8816a3e6-2fb9-4137-8db1-1fda30486a11-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:14.755252 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:14.755222    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"
Jul 31 19:15:14.755499 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:14.755217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk" event={"ID":"8816a3e6-2fb9-4137-8db1-1fda30486a11","Type":"ContainerDied","Data":"e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb"}
Jul 31 19:15:14.755499 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:14.755355    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34181a50cdf2b3b26595a582f04167aadfd295a9af083449d86aa5bb663b2eb"
Jul 31 19:15:18.052469 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.052423    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:15:18.052948 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.052859    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" podUID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" containerName="maas-api" containerID="cri-o://84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565" gracePeriod=30
Jul 31 19:15:18.115600 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.115565    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:15:18.116325 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.116287    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" podUID="9d544891-3b46-42f1-ac02-491239ff1d4f" containerName="payload-processing" containerID="cri-o://3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482" gracePeriod=30
Jul 31 19:15:18.125872 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.122181    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:15:18.125872 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.122394    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" podUID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" containerName="payload-pre-processing" containerID="cri-o://5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739" gracePeriod=30
Jul 31 19:15:18.261355 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.261319    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"]
Jul 31 19:15:18.264827 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.264783    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc42f1be-lmxjk"]
Jul 31 19:15:18.383580 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.383555    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:15:18.409371 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.408621    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:15:18.417611 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.417577    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:15:18.499133 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499102    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgg9p\" (UniqueName: \"kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p\") pod \"9d544891-3b46-42f1-ac02-491239ff1d4f\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lclkq\" (UniqueName: \"kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq\") pod \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499167    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdw85\" (UniqueName: \"kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85\") pod \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499195    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume\") pod \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\" (UID: \"3496ed0d-093d-4a3b-9cf0-0c4f438432da\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499227    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") pod \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499250    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca\") pod \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\" (UID: \"ea3aea66-7e6e-45d1-86e7-07c08d34122d\") "
Jul 31 19:15:18.499336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499267    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume\") pod \"9d544891-3b46-42f1-ac02-491239ff1d4f\" (UID: \"9d544891-3b46-42f1-ac02-491239ff1d4f\") "
Jul 31 19:15:18.499789 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499737    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea3aea66-7e6e-45d1-86e7-07c08d34122d" (UID: "ea3aea66-7e6e-45d1-86e7-07c08d34122d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:15:18.499789 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499745    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3496ed0d-093d-4a3b-9cf0-0c4f438432da" (UID: "3496ed0d-093d-4a3b-9cf0-0c4f438432da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:15:18.499963 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.499792    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d544891-3b46-42f1-ac02-491239ff1d4f" (UID: "9d544891-3b46-42f1-ac02-491239ff1d4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:15:18.501710 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.501679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea3aea66-7e6e-45d1-86e7-07c08d34122d" (UID: "ea3aea66-7e6e-45d1-86e7-07c08d34122d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:15:18.501821 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.501731    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85" (OuterVolumeSpecName: "kube-api-access-pdw85") pod "3496ed0d-093d-4a3b-9cf0-0c4f438432da" (UID: "3496ed0d-093d-4a3b-9cf0-0c4f438432da"). InnerVolumeSpecName "kube-api-access-pdw85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:15:18.502062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.501951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq" (OuterVolumeSpecName: "kube-api-access-lclkq") pod "ea3aea66-7e6e-45d1-86e7-07c08d34122d" (UID: "ea3aea66-7e6e-45d1-86e7-07c08d34122d"). InnerVolumeSpecName "kube-api-access-lclkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:15:18.502062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.502029    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p" (OuterVolumeSpecName: "kube-api-access-bgg9p") pod "9d544891-3b46-42f1-ac02-491239ff1d4f" (UID: "9d544891-3b46-42f1-ac02-491239ff1d4f"). InnerVolumeSpecName "kube-api-access-bgg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:15:18.600506 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600478    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgg9p\" (UniqueName: \"kubernetes.io/projected/9d544891-3b46-42f1-ac02-491239ff1d4f-kube-api-access-bgg9p\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600506 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600505    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lclkq\" (UniqueName: \"kubernetes.io/projected/ea3aea66-7e6e-45d1-86e7-07c08d34122d-kube-api-access-lclkq\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600706 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600517    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdw85\" (UniqueName: \"kubernetes.io/projected/3496ed0d-093d-4a3b-9cf0-0c4f438432da-kube-api-access-pdw85\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600706 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600526    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496ed0d-093d-4a3b-9cf0-0c4f438432da-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600706 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600536    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3aea66-7e6e-45d1-86e7-07c08d34122d-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600706 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600545    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3aea66-7e6e-45d1-86e7-07c08d34122d-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.600706 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.600553    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d544891-3b46-42f1-ac02-491239ff1d4f-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:15:18.775760 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.775724    2557 generic.go:356] "Generic (PLEG): container finished" podID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" containerID="5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739" exitCode=0
Jul 31 19:15:18.775760 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.775763    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"
Jul 31 19:15:18.775987 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.775805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" event={"ID":"3496ed0d-093d-4a3b-9cf0-0c4f438432da","Type":"ContainerDied","Data":"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"}
Jul 31 19:15:18.775987 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.775856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk" event={"ID":"3496ed0d-093d-4a3b-9cf0-0c4f438432da","Type":"ContainerDied","Data":"3ba79993d947b0ae8f9d0652d5d85bf1a05d40fe43789371fe558d762a8fb661"}
Jul 31 19:15:18.775987 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.775879    2557 scope.go:117] "RemoveContainer" containerID="5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"
Jul 31 19:15:18.777088 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.777064    2557 generic.go:356] "Generic (PLEG): container finished" podID="9d544891-3b46-42f1-ac02-491239ff1d4f" containerID="3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482" exitCode=0
Jul 31 19:15:18.777180 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.777092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" event={"ID":"9d544891-3b46-42f1-ac02-491239ff1d4f","Type":"ContainerDied","Data":"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"}
Jul 31 19:15:18.777180 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.777125    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"
Jul 31 19:15:18.777180 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.777140    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg" event={"ID":"9d544891-3b46-42f1-ac02-491239ff1d4f","Type":"ContainerDied","Data":"318791d72d234a765b2028a2e15333d0c9ecd071620e55fc2ebe421736b909a6"}
Jul 31 19:15:18.778568 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.778546    2557 generic.go:356] "Generic (PLEG): container finished" podID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" containerID="84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565" exitCode=0
Jul 31 19:15:18.778697 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.778572    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" event={"ID":"ea3aea66-7e6e-45d1-86e7-07c08d34122d","Type":"ContainerDied","Data":"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"}
Jul 31 19:15:18.778697 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.778592    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"
Jul 31 19:15:18.778697 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.778599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx" event={"ID":"ea3aea66-7e6e-45d1-86e7-07c08d34122d","Type":"ContainerDied","Data":"fa3a4a89e0002d21d10f48193c114e73d22da987cddee80805fa6b689bfb3dc2"}
Jul 31 19:15:18.786831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.786814    2557 scope.go:117] "RemoveContainer" containerID="5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"
Jul 31 19:15:18.787111 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:15:18.787089    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739\": container with ID starting with 5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739 not found: ID does not exist" containerID="5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"
Jul 31 19:15:18.787162 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.787120    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739"} err="failed to get container status \"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739\": rpc error: code = NotFound desc = could not find container \"5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739\": container with ID starting with 5cbb4b4b64a675c3164940980b53cc611b1f8a87dd7784ba625332569f430739 not found: ID does not exist"
Jul 31 19:15:18.787162 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.787136    2557 scope.go:117] "RemoveContainer" containerID="3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"
Jul 31 19:15:18.795677 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.795663    2557 scope.go:117] "RemoveContainer" containerID="3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"
Jul 31 19:15:18.795903 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:15:18.795885    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482\": container with ID starting with 3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482 not found: ID does not exist" containerID="3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"
Jul 31 19:15:18.795980 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.795912    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482"} err="failed to get container status \"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482\": rpc error: code = NotFound desc = could not find container \"3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482\": container with ID starting with 3bb9e88b7655df5ffd6d5e3a961add4b5f6f4bf4620a4d136ae1a3f586804482 not found: ID does not exist"
Jul 31 19:15:18.795980 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.795934    2557 scope.go:117] "RemoveContainer" containerID="84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"
Jul 31 19:15:18.804298 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.804279    2557 scope.go:117] "RemoveContainer" containerID="84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"
Jul 31 19:15:18.804622 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:15:18.804598    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565\": container with ID starting with 84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565 not found: ID does not exist" containerID="84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"
Jul 31 19:15:18.804714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.804627    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565"} err="failed to get container status \"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565\": rpc error: code = NotFound desc = could not find container \"84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565\": container with ID starting with 84647916bffad67a6f6c9150f22025d9f3f37a9c73ae539af6ec5b0579701565 not found: ID does not exist"
Jul 31 19:15:18.805682 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.805663    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:15:18.809925 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.809902    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc42f1be-6f6bb8b878-f2dsg"]
Jul 31 19:15:18.822754 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.822730    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:15:18.827270 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.827249    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc42f1be-7599d8fd74-rd2dx"]
Jul 31 19:15:18.840713 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.840691    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:15:18.845354 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:18.845334    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc42f1be-7485d9bc84-fg8sk"]
Jul 31 19:15:20.182776 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:20.182745    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" path="/var/lib/kubelet/pods/3496ed0d-093d-4a3b-9cf0-0c4f438432da/volumes"
Jul 31 19:15:20.183129 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:20.183065    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8816a3e6-2fb9-4137-8db1-1fda30486a11" path="/var/lib/kubelet/pods/8816a3e6-2fb9-4137-8db1-1fda30486a11/volumes"
Jul 31 19:15:20.183362 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:20.183350    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d544891-3b46-42f1-ac02-491239ff1d4f" path="/var/lib/kubelet/pods/9d544891-3b46-42f1-ac02-491239ff1d4f/volumes"
Jul 31 19:15:20.183700 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:15:20.183689    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" path="/var/lib/kubelet/pods/ea3aea66-7e6e-45d1-86e7-07c08d34122d/volumes"
Jul 31 19:17:23.873615 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.873574    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:17:23.874075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874019    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d544891-3b46-42f1-ac02-491239ff1d4f" containerName="payload-processing"
Jul 31 19:17:23.874075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874039    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d544891-3b46-42f1-ac02-491239ff1d4f" containerName="payload-processing"
Jul 31 19:17:23.874075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874052    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" containerName="maas-api"
Jul 31 19:17:23.874075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874061    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" containerName="maas-api"
Jul 31 19:17:23.874075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874074    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" containerName="payload-pre-processing"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874084    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" containerName="payload-pre-processing"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874116    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8816a3e6-2fb9-4137-8db1-1fda30486a11" containerName="revoke-keys"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874124    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8816a3e6-2fb9-4137-8db1-1fda30486a11" containerName="revoke-keys"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874194    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3aea66-7e6e-45d1-86e7-07c08d34122d" containerName="maas-api"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874208    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8816a3e6-2fb9-4137-8db1-1fda30486a11" containerName="revoke-keys"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874218    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3496ed0d-093d-4a3b-9cf0-0c4f438432da" containerName="payload-pre-processing"
Jul 31 19:17:23.874230 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.874229    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d544891-3b46-42f1-ac02-491239ff1d4f" containerName="payload-processing"
Jul 31 19:17:23.877341 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.877323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:23.881592 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.881573    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f249e039-serving-cert\""
Jul 31 19:17:23.881893 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.881871    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:17:23.882005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.881894    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:17:23.882005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.881930    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:17:23.888429 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:23.888377    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:17:24.039083 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.039046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.039265 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.039093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.039265 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.039174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l5r\" (UniqueName: \"kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.139831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.139738    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.139831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.139790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.139831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.139831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l5r\" (UniqueName: \"kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.140407 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.140381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.142386 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.142361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.152813 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.152794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l5r\" (UniqueName: \"kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r\") pod \"maas-api-e2e-mt-f249e039-86f7f56547-8t9s6\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.190225 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.190201    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:24.331254 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.331231    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:17:24.332837 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:17:24.332813    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode421cf97_e109_4aaa_97db_a642eeb665a4.slice/crio-b1e4a3fdc2afcf6c74fdf6ecce26e2fa5fb2e608081314dafa01d272368478da WatchSource:0}: Error finding container b1e4a3fdc2afcf6c74fdf6ecce26e2fa5fb2e608081314dafa01d272368478da: Status 404 returned error can't find the container with id b1e4a3fdc2afcf6c74fdf6ecce26e2fa5fb2e608081314dafa01d272368478da
Jul 31 19:17:24.371619 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.371579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" event={"ID":"e421cf97-e109-4aaa-97db-a642eeb665a4","Type":"ContainerStarted","Data":"b1e4a3fdc2afcf6c74fdf6ecce26e2fa5fb2e608081314dafa01d272368478da"}
Jul 31 19:17:24.720418 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.720388    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:17:24.724638 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.724616    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.727348 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.727330    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f249e039\""
Jul 31 19:17:24.727489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.727461    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f249e039-dockercfg-4c5bm\""
Jul 31 19:17:24.734692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.734670    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:17:24.737675 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.737653    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:17:24.737801 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.737763    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.751139 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.751115    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:17:24.844819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.844782    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.844819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.844824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.845026 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.844924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md27t\" (UniqueName: \"kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.845026 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.844977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9rr\" (UniqueName: \"kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.945882 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.945844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9rr\" (UniqueName: \"kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.946290 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.945909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.946290 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.945934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.946290 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.945969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md27t\" (UniqueName: \"kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.946703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.946681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.946755 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.946720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:24.955488 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.955436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md27t\" (UniqueName: \"kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t\") pod \"payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:24.955594 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:24.955500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9rr\" (UniqueName: \"kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr\") pod \"payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:25.036193 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.036111    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:25.048484 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.048433    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:25.182970 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.182936    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:17:25.185383 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:17:25.185354    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd42572_ef2b_4d50_b588_c8623ea6b6fd.slice/crio-25b0de9c01320039a7d0dfd1abe8e0002b0ce8bb93c69d240e31c34519142fde WatchSource:0}: Error finding container 25b0de9c01320039a7d0dfd1abe8e0002b0ce8bb93c69d240e31c34519142fde: Status 404 returned error can't find the container with id 25b0de9c01320039a7d0dfd1abe8e0002b0ce8bb93c69d240e31c34519142fde
Jul 31 19:17:25.204494 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.204465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:17:25.206404 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:17:25.206271    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618ea567_78da_4d42_a5a0_ffb3b47d45d5.slice/crio-672a4742a159741543eb7c57000cd2cd8ea2b105e861682fe3ee0376ca722ea6 WatchSource:0}: Error finding container 672a4742a159741543eb7c57000cd2cd8ea2b105e861682fe3ee0376ca722ea6: Status 404 returned error can't find the container with id 672a4742a159741543eb7c57000cd2cd8ea2b105e861682fe3ee0376ca722ea6
Jul 31 19:17:25.379343 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.379310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" event={"ID":"cfd42572-ef2b-4d50-b588-c8623ea6b6fd","Type":"ContainerStarted","Data":"de1d2d6ab0a16d272a3d87e608afbca5173b534794b8ddb818c106d3e4126b1d"}
Jul 31 19:17:25.379572 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.379373    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" event={"ID":"cfd42572-ef2b-4d50-b588-c8623ea6b6fd","Type":"ContainerStarted","Data":"25b0de9c01320039a7d0dfd1abe8e0002b0ce8bb93c69d240e31c34519142fde"}
Jul 31 19:17:25.379572 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.379403    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:25.380946 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.380923    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" event={"ID":"e421cf97-e109-4aaa-97db-a642eeb665a4","Type":"ContainerStarted","Data":"f2cd3d775fdda003e0d82ae6520802ef3e26ca0c0d2806b87911b470e4d8e98f"}
Jul 31 19:17:25.381081 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.381062    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:25.382513 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.382488    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" event={"ID":"618ea567-78da-4d42-a5a0-ffb3b47d45d5","Type":"ContainerStarted","Data":"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"}
Jul 31 19:17:25.382608 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.382534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" event={"ID":"618ea567-78da-4d42-a5a0-ffb3b47d45d5","Type":"ContainerStarted","Data":"672a4742a159741543eb7c57000cd2cd8ea2b105e861682fe3ee0376ca722ea6"}
Jul 31 19:17:25.382608 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.382562    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:25.399129 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.399086    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" podStartSLOduration=1.399071435 podStartE2EDuration="1.399071435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:17:25.397305383 +0000 UTC m=+2023.744887358" watchObservedRunningTime="2026-07-31 19:17:25.399071435 +0000 UTC m=+2023.746653293"
Jul 31 19:17:25.416139 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.416083    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" podStartSLOduration=1.903881944 podStartE2EDuration="2.416068689s" totalImagesPullingTime="512.186745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:17:25.414787377 +0000 UTC m=+2023.762369266" watchObservedRunningTime="2026-07-31 19:17:25.416068689 +0000 UTC m=+2023.763650568"
Jul 31 19:17:25.433774 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:25.433724    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" podStartSLOduration=1.433708245 podStartE2EDuration="1.433708245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:17:25.432673005 +0000 UTC m=+2023.780254886" watchObservedRunningTime="2026-07-31 19:17:25.433708245 +0000 UTC m=+2023.781290125"
Jul 31 19:17:31.393653 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:31.393622    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:17:36.389330 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:36.389299    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:17:36.389840 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:36.389382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:17:55.437970 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.437929    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"]
Jul 31 19:17:55.441341 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.441321    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:55.443794 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.443772    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:17:55.449311 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.449286    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"]
Jul 31 19:17:55.504747 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.504709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f249e039-sj9x9\" (UID: \"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:55.605950 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.605916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f249e039-sj9x9\" (UID: \"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:55.606605 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.606587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f249e039-sj9x9\" (UID: \"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:55.753395 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.753311    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:55.885957 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:55.885931    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"]
Jul 31 19:17:55.887735 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:17:55.887709    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b17ce1_0fa1_4fe1_b56b_5d7d0efb4a86.slice/crio-24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913 WatchSource:0}: Error finding container 24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913: Status 404 returned error can't find the container with id 24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913
Jul 31 19:17:56.539617 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:56.539578    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9" event={"ID":"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86","Type":"ContainerStarted","Data":"0a75f2089a1b256d7a8d6519bbb756dadc0dcd38d07844b3d9b3b69c158d6cc7"}
Jul 31 19:17:56.540022 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:56.539642    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9" event={"ID":"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86","Type":"ContainerStarted","Data":"24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913"}
Jul 31 19:17:56.557321 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:56.557268    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9" podStartSLOduration=1.557251629 podStartE2EDuration="1.557251629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:17:56.5566748 +0000 UTC m=+2054.904256682" watchObservedRunningTime="2026-07-31 19:17:56.557251629 +0000 UTC m=+2054.904833509"
Jul 31 19:17:57.550738 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:57.550699    2557 generic.go:356] "Generic (PLEG): container finished" podID="30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" containerID="0a75f2089a1b256d7a8d6519bbb756dadc0dcd38d07844b3d9b3b69c158d6cc7" exitCode=0
Jul 31 19:17:57.551143 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:57.550767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9" event={"ID":"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86","Type":"ContainerDied","Data":"0a75f2089a1b256d7a8d6519bbb756dadc0dcd38d07844b3d9b3b69c158d6cc7"}
Jul 31 19:17:58.685748 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:58.685712    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:58.833358 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:58.833272    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca\") pod \"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86\" (UID: \"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86\") "
Jul 31 19:17:58.833672 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:58.833647    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" (UID: "30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:17:58.934153 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:58.934116    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:17:59.562311 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:59.562281    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"
Jul 31 19:17:59.562311 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:59.562295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9" event={"ID":"30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86","Type":"ContainerDied","Data":"24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913"}
Jul 31 19:17:59.562554 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:17:59.562343    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24cbcd352eafd81a622bf7d577128d45373ea8e3b20f127ca13950b1bac5b913"
Jul 31 19:18:06.258742 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.258663    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:18:06.259304 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.258864    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerName="maas-api" containerID="cri-o://f2cd3d775fdda003e0d82ae6520802ef3e26ca0c0d2806b87911b470e4d8e98f" gracePeriod=30
Jul 31 19:18:06.320918 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.320876    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:18:06.321167 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.321140    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerName="payload-processing" containerID="cri-o://e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375" gracePeriod=30
Jul 31 19:18:06.324995 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.324967    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:18:06.325289 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.325259    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerName="payload-pre-processing" containerID="cri-o://de1d2d6ab0a16d272a3d87e608afbca5173b534794b8ddb818c106d3e4126b1d" gracePeriod=30
Jul 31 19:18:06.387958 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.387910    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.45:9004: connect: connection refused"
Jul 31 19:18:06.388126 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.387953    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.46:9004: connect: connection refused"
Jul 31 19:18:06.388126 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.387964    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.44:8443/health\": dial tcp 10.132.0.44:8443: connect: connection refused"
Jul 31 19:18:06.479432 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.479319    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"]
Jul 31 19:18:06.481567 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.481537    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f249e039-sj9x9"]
Jul 31 19:18:06.596094 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.596068    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:18:06.598847 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.598825    2557 generic.go:356] "Generic (PLEG): container finished" podID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerID="de1d2d6ab0a16d272a3d87e608afbca5173b534794b8ddb818c106d3e4126b1d" exitCode=0
Jul 31 19:18:06.598983 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.598896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" event={"ID":"cfd42572-ef2b-4d50-b588-c8623ea6b6fd","Type":"ContainerDied","Data":"de1d2d6ab0a16d272a3d87e608afbca5173b534794b8ddb818c106d3e4126b1d"}
Jul 31 19:18:06.600550 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.600527    2557 generic.go:356] "Generic (PLEG): container finished" podID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerID="f2cd3d775fdda003e0d82ae6520802ef3e26ca0c0d2806b87911b470e4d8e98f" exitCode=0
Jul 31 19:18:06.600643 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.600599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" event={"ID":"e421cf97-e109-4aaa-97db-a642eeb665a4","Type":"ContainerDied","Data":"f2cd3d775fdda003e0d82ae6520802ef3e26ca0c0d2806b87911b470e4d8e98f"}
Jul 31 19:18:06.601851 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.601835    2557 generic.go:356] "Generic (PLEG): container finished" podID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerID="e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375" exitCode=0
Jul 31 19:18:06.601943 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.601895    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" event={"ID":"618ea567-78da-4d42-a5a0-ffb3b47d45d5","Type":"ContainerDied","Data":"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"}
Jul 31 19:18:06.601943 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.601930    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"
Jul 31 19:18:06.601943 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.601937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h" event={"ID":"618ea567-78da-4d42-a5a0-ffb3b47d45d5","Type":"ContainerDied","Data":"672a4742a159741543eb7c57000cd2cd8ea2b105e861682fe3ee0376ca722ea6"}
Jul 31 19:18:06.602093 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.601958    2557 scope.go:117] "RemoveContainer" containerID="e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"
Jul 31 19:18:06.609140 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.609118    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:18:06.611161 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.611136    2557 scope.go:117] "RemoveContainer" containerID="e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"
Jul 31 19:18:06.611541 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:18:06.611503    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375\": container with ID starting with e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375 not found: ID does not exist" containerID="e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"
Jul 31 19:18:06.611599 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.611539    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375"} err="failed to get container status \"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375\": rpc error: code = NotFound desc = could not find container \"e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375\": container with ID starting with e43aaa379181e1dd284750e0ee5e583e4a340a58c6ed43ce13f966756fd30375 not found: ID does not exist"
Jul 31 19:18:06.620703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.620681    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:18:06.700819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls\") pod \"e421cf97-e109-4aaa-97db-a642eeb665a4\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") "
Jul 31 19:18:06.701000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700834    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv9rr\" (UniqueName: \"kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr\") pod \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") "
Jul 31 19:18:06.701000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume\") pod \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") "
Jul 31 19:18:06.701000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29l5r\" (UniqueName: \"kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r\") pod \"e421cf97-e109-4aaa-97db-a642eeb665a4\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") "
Jul 31 19:18:06.701000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700945    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume\") pod \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\" (UID: \"618ea567-78da-4d42-a5a0-ffb3b47d45d5\") "
Jul 31 19:18:06.701000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.700972    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca\") pod \"e421cf97-e109-4aaa-97db-a642eeb665a4\" (UID: \"e421cf97-e109-4aaa-97db-a642eeb665a4\") "
Jul 31 19:18:06.701211 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.701026    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md27t\" (UniqueName: \"kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t\") pod \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\" (UID: \"cfd42572-ef2b-4d50-b588-c8623ea6b6fd\") "
Jul 31 19:18:06.701372 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.701342    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfd42572-ef2b-4d50-b588-c8623ea6b6fd" (UID: "cfd42572-ef2b-4d50-b588-c8623ea6b6fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:18:06.701372 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.701360    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "618ea567-78da-4d42-a5a0-ffb3b47d45d5" (UID: "618ea567-78da-4d42-a5a0-ffb3b47d45d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:18:06.701680 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.701614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e421cf97-e109-4aaa-97db-a642eeb665a4" (UID: "e421cf97-e109-4aaa-97db-a642eeb665a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:18:06.703310 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.703288    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r" (OuterVolumeSpecName: "kube-api-access-29l5r") pod "e421cf97-e109-4aaa-97db-a642eeb665a4" (UID: "e421cf97-e109-4aaa-97db-a642eeb665a4"). InnerVolumeSpecName "kube-api-access-29l5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:18:06.703388 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.703310    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t" (OuterVolumeSpecName: "kube-api-access-md27t") pod "cfd42572-ef2b-4d50-b588-c8623ea6b6fd" (UID: "cfd42572-ef2b-4d50-b588-c8623ea6b6fd"). InnerVolumeSpecName "kube-api-access-md27t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:18:06.703630 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.703603    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e421cf97-e109-4aaa-97db-a642eeb665a4" (UID: "e421cf97-e109-4aaa-97db-a642eeb665a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:18:06.703630 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.703608    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr" (OuterVolumeSpecName: "kube-api-access-sv9rr") pod "618ea567-78da-4d42-a5a0-ffb3b47d45d5" (UID: "618ea567-78da-4d42-a5a0-ffb3b47d45d5"). InnerVolumeSpecName "kube-api-access-sv9rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801588    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/618ea567-78da-4d42-a5a0-ffb3b47d45d5-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801613    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e421cf97-e109-4aaa-97db-a642eeb665a4-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801625    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md27t\" (UniqueName: \"kubernetes.io/projected/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-kube-api-access-md27t\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801634    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e421cf97-e109-4aaa-97db-a642eeb665a4-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801643    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9rr\" (UniqueName: \"kubernetes.io/projected/618ea567-78da-4d42-a5a0-ffb3b47d45d5-kube-api-access-sv9rr\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801651    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd42572-ef2b-4d50-b588-c8623ea6b6fd-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.801664 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.801659    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29l5r\" (UniqueName: \"kubernetes.io/projected/e421cf97-e109-4aaa-97db-a642eeb665a4-kube-api-access-29l5r\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:18:06.925967 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.925934    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:18:06.933201 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:06.933154    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f249e039-779cd9579d-f2r6h"]
Jul 31 19:18:07.609460 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.609407    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"
Jul 31 19:18:07.609460 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.609431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj" event={"ID":"cfd42572-ef2b-4d50-b588-c8623ea6b6fd","Type":"ContainerDied","Data":"25b0de9c01320039a7d0dfd1abe8e0002b0ce8bb93c69d240e31c34519142fde"}
Jul 31 19:18:07.609975 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.609515    2557 scope.go:117] "RemoveContainer" containerID="de1d2d6ab0a16d272a3d87e608afbca5173b534794b8ddb818c106d3e4126b1d"
Jul 31 19:18:07.610817 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.610797    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6" event={"ID":"e421cf97-e109-4aaa-97db-a642eeb665a4","Type":"ContainerDied","Data":"b1e4a3fdc2afcf6c74fdf6ecce26e2fa5fb2e608081314dafa01d272368478da"}
Jul 31 19:18:07.610896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.610852    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"
Jul 31 19:18:07.618551 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.618533    2557 scope.go:117] "RemoveContainer" containerID="f2cd3d775fdda003e0d82ae6520802ef3e26ca0c0d2806b87911b470e4d8e98f"
Jul 31 19:18:07.636775 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.636752    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:18:07.642118 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.642097    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f249e039-86f7f56547-8t9s6"]
Jul 31 19:18:07.654213 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.654192    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:18:07.659326 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:07.659305    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f249e039-78d986644d-p4dnj"]
Jul 31 19:18:08.182708 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:08.182678    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" path="/var/lib/kubelet/pods/30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86/volumes"
Jul 31 19:18:08.183095 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:08.183079    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" path="/var/lib/kubelet/pods/618ea567-78da-4d42-a5a0-ffb3b47d45d5/volumes"
Jul 31 19:18:08.183428 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:08.183416    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" path="/var/lib/kubelet/pods/cfd42572-ef2b-4d50-b588-c8623ea6b6fd/volumes"
Jul 31 19:18:08.183821 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:08.183807    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" path="/var/lib/kubelet/pods/e421cf97-e109-4aaa-97db-a642eeb665a4/volumes"
Jul 31 19:18:42.266609 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:42.266581    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:18:42.268623 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:18:42.268602    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:19:42.356271 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:19:42.356203    2557 scope.go:117] "RemoveContainer" containerID="709bc03c3978912eea93efe3e5ced6767cafc69fa278ef5025213b5bc44c5809"
Jul 31 19:20:33.852766 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.852731    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:20:33.853200 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853174    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerName="payload-processing"
Jul 31 19:20:33.853200 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853201    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerName="payload-processing"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853245    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerName="payload-pre-processing"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853255    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerName="payload-pre-processing"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853268    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerName="maas-api"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853276    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerName="maas-api"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853286    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" containerName="revoke-keys"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853295    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" containerName="revoke-keys"
Jul 31 19:20:33.853399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853393    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b17ce1-0fa1-4fe1-b56b-5d7d0efb4a86" containerName="revoke-keys"
Jul 31 19:20:33.853683 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853407    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd42572-ef2b-4d50-b588-c8623ea6b6fd" containerName="payload-pre-processing"
Jul 31 19:20:33.853683 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853418    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e421cf97-e109-4aaa-97db-a642eeb665a4" containerName="maas-api"
Jul 31 19:20:33.853683 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.853432    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="618ea567-78da-4d42-a5a0-ffb3b47d45d5" containerName="payload-processing"
Jul 31 19:20:33.856651 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.856633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:33.859223 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.859192    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:20:33.859357 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.859331    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aaeacbf1-serving-cert\""
Jul 31 19:20:33.859357 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.859342    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:20:33.860326 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.860305    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:20:33.866627 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.866605    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:20:33.916688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.916648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzz8\" (UniqueName: \"kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:33.916869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.916712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:33.916869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:33.916730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.017834 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.017792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzz8\" (UniqueName: \"kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.018040 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.017859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.018040 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.017888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.018040 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:20:34.017957    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-serving-cert: secret "maas-api-e2e-mt-aaeacbf1-serving-cert" not found
Jul 31 19:20:34.018040 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:20:34.018028    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls podName:2cfd63e1-ea14-4524-b032-de269edeef12 nodeName:}" failed. No retries permitted until 2026-07-31 19:20:34.518012049 +0000 UTC m=+2212.865593910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls") pod "maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" (UID: "2cfd63e1-ea14-4524-b032-de269edeef12") : secret "maas-api-e2e-mt-aaeacbf1-serving-cert" not found
Jul 31 19:20:34.018897 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.018872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.031273 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.031243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzz8\" (UniqueName: \"kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.523097 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.523059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.525754 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.525729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") pod \"maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.552708 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.552679    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:20:34.555939 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.555925    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.558633 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.558611    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aaeacbf1-dockercfg-jjxls\""
Jul 31 19:20:34.558759 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.558651    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aaeacbf1\""
Jul 31 19:20:34.570908 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.570885    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:20:34.598604 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.598573    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:20:34.601718 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.601704    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.612606 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.612579    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:20:34.623810 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.623779    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pn7\" (UniqueName: \"kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.623981 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.623854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.623981 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.623895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmj9\" (UniqueName: \"kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.624062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.623972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.724369 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.724334    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmj9\" (UniqueName: \"kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.724618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.724388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.724618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.724406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pn7\" (UniqueName: \"kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.724618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.724488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.725196 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.725171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.725288 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.725179    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.737294 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.737274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pn7\" (UniqueName: \"kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7\") pod \"payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:34.737385 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.737316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmj9\" (UniqueName: \"kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9\") pod \"payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") " pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.767901 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.767879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:34.865791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.865755    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:34.903508 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.903480    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:20:34.905010 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:20:34.904988    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfd63e1_ea14_4524_b032_de269edeef12.slice/crio-1ea7b991a8d7dfdadd593e4c3646ba2856a06ceed304bb69f03247657fbd582c WatchSource:0}: Error finding container 1ea7b991a8d7dfdadd593e4c3646ba2856a06ceed304bb69f03247657fbd582c: Status 404 returned error can't find the container with id 1ea7b991a8d7dfdadd593e4c3646ba2856a06ceed304bb69f03247657fbd582c
Jul 31 19:20:34.906727 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.906711    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 19:20:34.912518 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:34.912488    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:35.009179 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.009125    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:20:35.010023 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:20:35.009987    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147183f7_388f_4aed_8bcf_1a37698154a4.slice/crio-e801988fb90bbcdc0d01ec7a3a7414fa5e31fca8ba4d4dfbafa1f79abcf7dd5e WatchSource:0}: Error finding container e801988fb90bbcdc0d01ec7a3a7414fa5e31fca8ba4d4dfbafa1f79abcf7dd5e: Status 404 returned error can't find the container with id e801988fb90bbcdc0d01ec7a3a7414fa5e31fca8ba4d4dfbafa1f79abcf7dd5e
Jul 31 19:20:35.060291 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.060212    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:20:35.062744 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:20:35.062719    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526efde7_5d67_4858_8b32_9cbae4bba0f0.slice/crio-edc76bb7458a167e735dfed62a8b2cdc779aa1f68d6af807a73b7afbe82e11a4 WatchSource:0}: Error finding container edc76bb7458a167e735dfed62a8b2cdc779aa1f68d6af807a73b7afbe82e11a4: Status 404 returned error can't find the container with id edc76bb7458a167e735dfed62a8b2cdc779aa1f68d6af807a73b7afbe82e11a4
Jul 31 19:20:35.327124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.327042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" event={"ID":"526efde7-5d67-4858-8b32-9cbae4bba0f0","Type":"ContainerStarted","Data":"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"}
Jul 31 19:20:35.327124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.327094    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" event={"ID":"526efde7-5d67-4858-8b32-9cbae4bba0f0","Type":"ContainerStarted","Data":"edc76bb7458a167e735dfed62a8b2cdc779aa1f68d6af807a73b7afbe82e11a4"}
Jul 31 19:20:35.327350 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.327128    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:20:35.328666 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.328636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" event={"ID":"147183f7-388f-4aed-8bcf-1a37698154a4","Type":"ContainerStarted","Data":"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"}
Jul 31 19:20:35.328817 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.328687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" event={"ID":"147183f7-388f-4aed-8bcf-1a37698154a4","Type":"ContainerStarted","Data":"e801988fb90bbcdc0d01ec7a3a7414fa5e31fca8ba4d4dfbafa1f79abcf7dd5e"}
Jul 31 19:20:35.328817 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.328720    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:35.329774 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.329755    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" event={"ID":"2cfd63e1-ea14-4524-b032-de269edeef12","Type":"ContainerStarted","Data":"1ea7b991a8d7dfdadd593e4c3646ba2856a06ceed304bb69f03247657fbd582c"}
Jul 31 19:20:35.347430 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.347208    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" podStartSLOduration=1.347189786 podStartE2EDuration="1.347189786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:20:35.346687828 +0000 UTC m=+2213.694269710" watchObservedRunningTime="2026-07-31 19:20:35.347189786 +0000 UTC m=+2213.694771667"
Jul 31 19:20:35.366341 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:35.366298    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" podStartSLOduration=1.366283464 podStartE2EDuration="1.366283464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:20:35.364866412 +0000 UTC m=+2213.712448295" watchObservedRunningTime="2026-07-31 19:20:35.366283464 +0000 UTC m=+2213.713865379"
Jul 31 19:20:36.336695 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:36.336653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" event={"ID":"2cfd63e1-ea14-4524-b032-de269edeef12","Type":"ContainerStarted","Data":"5eeb27467fa8c4200ca6e1a5456c7196334b2dbbb6c921591a932d41e222400c"}
Jul 31 19:20:36.356117 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:36.355936    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" podStartSLOduration=2.9304822870000002 podStartE2EDuration="3.355918604s" totalImagesPullingTime="425.436317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:20:36.355286328 +0000 UTC m=+2214.702868208" watchObservedRunningTime="2026-07-31 19:20:36.355918604 +0000 UTC m=+2214.703500485"
Jul 31 19:20:37.341890 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:37.341862    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:42.374153 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:42.374126    2557 scope.go:117] "RemoveContainer" containerID="4d1258a0918194371c72fd82877fe44750e7871190e8723c354217049064830d"
Jul 31 19:20:43.354520 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:43.354488    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:20:46.339098 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:46.339066    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:20:46.339512 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:20:46.339136    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:21:31.839568 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:31.839531    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"]
Jul 31 19:21:31.842777 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:31.842761    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:31.845352 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:31.845333    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:21:31.852581 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:31.852556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"]
Jul 31 19:21:31.905833 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:31.905800    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b\" (UID: \"edb9098f-451f-4d3a-a44a-065357f43615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:32.006838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.006800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b\" (UID: \"edb9098f-451f-4d3a-a44a-065357f43615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:32.007543 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.007522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b\" (UID: \"edb9098f-451f-4d3a-a44a-065357f43615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:32.153562 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.153525    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:32.288907 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.288881    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"]
Jul 31 19:21:32.290427 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:21:32.290396    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb9098f_451f_4d3a_a44a_065357f43615.slice/crio-73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619 WatchSource:0}: Error finding container 73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619: Status 404 returned error can't find the container with id 73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619
Jul 31 19:21:32.609489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.609433    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b" event={"ID":"edb9098f-451f-4d3a-a44a-065357f43615","Type":"ContainerStarted","Data":"a650dd00b4a9fedfa6d2a8443571258bc10bd4596611029dcdc7ca83efcd37be"}
Jul 31 19:21:32.609699 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.609503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b" event={"ID":"edb9098f-451f-4d3a-a44a-065357f43615","Type":"ContainerStarted","Data":"73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619"}
Jul 31 19:21:32.627196 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:32.627137    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b" podStartSLOduration=1.627123136 podStartE2EDuration="1.627123136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:21:32.625954812 +0000 UTC m=+2270.973536694" watchObservedRunningTime="2026-07-31 19:21:32.627123136 +0000 UTC m=+2270.974705015"
Jul 31 19:21:33.616246 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:33.616209    2557 generic.go:356] "Generic (PLEG): container finished" podID="edb9098f-451f-4d3a-a44a-065357f43615" containerID="a650dd00b4a9fedfa6d2a8443571258bc10bd4596611029dcdc7ca83efcd37be" exitCode=0
Jul 31 19:21:33.616677 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:33.616296    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b" event={"ID":"edb9098f-451f-4d3a-a44a-065357f43615","Type":"ContainerDied","Data":"a650dd00b4a9fedfa6d2a8443571258bc10bd4596611029dcdc7ca83efcd37be"}
Jul 31 19:21:34.745627 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:34.745603    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:34.832354 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:34.832319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca\") pod \"edb9098f-451f-4d3a-a44a-065357f43615\" (UID: \"edb9098f-451f-4d3a-a44a-065357f43615\") "
Jul 31 19:21:34.832701 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:34.832677    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edb9098f-451f-4d3a-a44a-065357f43615" (UID: "edb9098f-451f-4d3a-a44a-065357f43615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:21:34.933371 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:34.933280    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb9098f-451f-4d3a-a44a-065357f43615-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:35.627887 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:35.627853    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"
Jul 31 19:21:35.628065 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:35.627901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b" event={"ID":"edb9098f-451f-4d3a-a44a-065357f43615","Type":"ContainerDied","Data":"73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619"}
Jul 31 19:21:35.628065 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:35.627951    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b2aa2cc091094627bfbf7941af978d3ab5a6c4204586dba5471ed8ea2af619"
Jul 31 19:21:42.393009 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:42.392973    2557 scope.go:117] "RemoveContainer" containerID="6c45677ef4a0ca39205166b0c08acdbec988bd70a42b1de637a240964d9c0381"
Jul 31 19:21:47.468960 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.468928    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:21:47.469416 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.469115    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" podUID="2cfd63e1-ea14-4524-b032-de269edeef12" containerName="maas-api" containerID="cri-o://5eeb27467fa8c4200ca6e1a5456c7196334b2dbbb6c921591a932d41e222400c" gracePeriod=30
Jul 31 19:21:47.608742 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.608707    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:21:47.609062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.608999    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" podUID="147183f7-388f-4aed-8bcf-1a37698154a4" containerName="payload-processing" containerID="cri-o://a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92" gracePeriod=30
Jul 31 19:21:47.621529 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.621201    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:21:47.621529 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.621478    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" podUID="526efde7-5d67-4858-8b32-9cbae4bba0f0" containerName="payload-pre-processing" containerID="cri-o://68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804" gracePeriod=30
Jul 31 19:21:47.692228 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.692199    2557 generic.go:356] "Generic (PLEG): container finished" podID="2cfd63e1-ea14-4524-b032-de269edeef12" containerID="5eeb27467fa8c4200ca6e1a5456c7196334b2dbbb6c921591a932d41e222400c" exitCode=0
Jul 31 19:21:47.692353 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.692246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" event={"ID":"2cfd63e1-ea14-4524-b032-de269edeef12","Type":"ContainerDied","Data":"5eeb27467fa8c4200ca6e1a5456c7196334b2dbbb6c921591a932d41e222400c"}
Jul 31 19:21:47.745425 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.745399    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:21:47.808791 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.808751    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"]
Jul 31 19:21:47.811172 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.811149    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-aaeacbf1-lnq2b"]
Jul 31 19:21:47.843150 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.843078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca\") pod \"2cfd63e1-ea14-4524-b032-de269edeef12\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") "
Jul 31 19:21:47.843303 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.843187    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") pod \"2cfd63e1-ea14-4524-b032-de269edeef12\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") "
Jul 31 19:21:47.843303 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.843254    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrzz8\" (UniqueName: \"kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8\") pod \"2cfd63e1-ea14-4524-b032-de269edeef12\" (UID: \"2cfd63e1-ea14-4524-b032-de269edeef12\") "
Jul 31 19:21:47.843515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.843487    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2cfd63e1-ea14-4524-b032-de269edeef12" (UID: "2cfd63e1-ea14-4524-b032-de269edeef12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:21:47.845533 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.845497    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8" (OuterVolumeSpecName: "kube-api-access-xrzz8") pod "2cfd63e1-ea14-4524-b032-de269edeef12" (UID: "2cfd63e1-ea14-4524-b032-de269edeef12"). InnerVolumeSpecName "kube-api-access-xrzz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:21:47.845867 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.845843    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2cfd63e1-ea14-4524-b032-de269edeef12" (UID: "2cfd63e1-ea14-4524-b032-de269edeef12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:21:47.882675 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.882650    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:21:47.886844 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.886822    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:21:47.944754 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944719    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8pn7\" (UniqueName: \"kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7\") pod \"526efde7-5d67-4858-8b32-9cbae4bba0f0\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") "
Jul 31 19:21:47.944950 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944770    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume\") pod \"147183f7-388f-4aed-8bcf-1a37698154a4\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") "
Jul 31 19:21:47.944950 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zmj9\" (UniqueName: \"kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9\") pod \"147183f7-388f-4aed-8bcf-1a37698154a4\" (UID: \"147183f7-388f-4aed-8bcf-1a37698154a4\") "
Jul 31 19:21:47.944950 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944846    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume\") pod \"526efde7-5d67-4858-8b32-9cbae4bba0f0\" (UID: \"526efde7-5d67-4858-8b32-9cbae4bba0f0\") "
Jul 31 19:21:47.945115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944965    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfd63e1-ea14-4524-b032-de269edeef12-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:47.945115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944979    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cfd63e1-ea14-4524-b032-de269edeef12-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:47.945115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.944993    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzz8\" (UniqueName: \"kubernetes.io/projected/2cfd63e1-ea14-4524-b032-de269edeef12-kube-api-access-xrzz8\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:47.945243 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.945221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "147183f7-388f-4aed-8bcf-1a37698154a4" (UID: "147183f7-388f-4aed-8bcf-1a37698154a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:21:47.945280 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.945240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "526efde7-5d67-4858-8b32-9cbae4bba0f0" (UID: "526efde7-5d67-4858-8b32-9cbae4bba0f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:21:47.947022 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.946999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9" (OuterVolumeSpecName: "kube-api-access-7zmj9") pod "147183f7-388f-4aed-8bcf-1a37698154a4" (UID: "147183f7-388f-4aed-8bcf-1a37698154a4"). InnerVolumeSpecName "kube-api-access-7zmj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:21:47.947111 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:47.947066    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7" (OuterVolumeSpecName: "kube-api-access-w8pn7") pod "526efde7-5d67-4858-8b32-9cbae4bba0f0" (UID: "526efde7-5d67-4858-8b32-9cbae4bba0f0"). InnerVolumeSpecName "kube-api-access-w8pn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:21:48.046142 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.046060    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmj9\" (UniqueName: \"kubernetes.io/projected/147183f7-388f-4aed-8bcf-1a37698154a4-kube-api-access-7zmj9\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:48.046142 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.046090    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526efde7-5d67-4858-8b32-9cbae4bba0f0-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:48.046142 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.046099    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pn7\" (UniqueName: \"kubernetes.io/projected/526efde7-5d67-4858-8b32-9cbae4bba0f0-kube-api-access-w8pn7\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:48.046142 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.046129    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147183f7-388f-4aed-8bcf-1a37698154a4-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:21:48.182801 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.182769    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb9098f-451f-4d3a-a44a-065357f43615" path="/var/lib/kubelet/pods/edb9098f-451f-4d3a-a44a-065357f43615/volumes"
Jul 31 19:21:48.699212 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.699176    2557 generic.go:356] "Generic (PLEG): container finished" podID="147183f7-388f-4aed-8bcf-1a37698154a4" containerID="a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92" exitCode=0
Jul 31 19:21:48.699671 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.699254    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"
Jul 31 19:21:48.699671 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.699276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" event={"ID":"147183f7-388f-4aed-8bcf-1a37698154a4","Type":"ContainerDied","Data":"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"}
Jul 31 19:21:48.699671 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.699362    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j" event={"ID":"147183f7-388f-4aed-8bcf-1a37698154a4","Type":"ContainerDied","Data":"e801988fb90bbcdc0d01ec7a3a7414fa5e31fca8ba4d4dfbafa1f79abcf7dd5e"}
Jul 31 19:21:48.699671 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.699389    2557 scope.go:117] "RemoveContainer" containerID="a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"
Jul 31 19:21:48.700926 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.700904    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"
Jul 31 19:21:48.701055 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.700989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68" event={"ID":"2cfd63e1-ea14-4524-b032-de269edeef12","Type":"ContainerDied","Data":"1ea7b991a8d7dfdadd593e4c3646ba2856a06ceed304bb69f03247657fbd582c"}
Jul 31 19:21:48.702434 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.702411    2557 generic.go:356] "Generic (PLEG): container finished" podID="526efde7-5d67-4858-8b32-9cbae4bba0f0" containerID="68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804" exitCode=0
Jul 31 19:21:48.702561 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.702467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" event={"ID":"526efde7-5d67-4858-8b32-9cbae4bba0f0","Type":"ContainerDied","Data":"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"}
Jul 31 19:21:48.702561 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.702502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f" event={"ID":"526efde7-5d67-4858-8b32-9cbae4bba0f0","Type":"ContainerDied","Data":"edc76bb7458a167e735dfed62a8b2cdc779aa1f68d6af807a73b7afbe82e11a4"}
Jul 31 19:21:48.702561 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.702506    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"
Jul 31 19:21:48.709545 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.709527    2557 scope.go:117] "RemoveContainer" containerID="a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"
Jul 31 19:21:48.709846 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:21:48.709825    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92\": container with ID starting with a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92 not found: ID does not exist" containerID="a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"
Jul 31 19:21:48.709928 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.709858    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92"} err="failed to get container status \"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92\": rpc error: code = NotFound desc = could not find container \"a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92\": container with ID starting with a910ada6f55b341b3309220ca3c027b6387f46f43ea722ac8e68f25ab472cc92 not found: ID does not exist"
Jul 31 19:21:48.709928 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.709883    2557 scope.go:117] "RemoveContainer" containerID="5eeb27467fa8c4200ca6e1a5456c7196334b2dbbb6c921591a932d41e222400c"
Jul 31 19:21:48.718069 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.718049    2557 scope.go:117] "RemoveContainer" containerID="68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"
Jul 31 19:21:48.723255 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.723231    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:21:48.727743 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.727715    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aaeacbf1-754b5f84cb-prs7f"]
Jul 31 19:21:48.729081 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.729056    2557 scope.go:117] "RemoveContainer" containerID="68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"
Jul 31 19:21:48.729377 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:21:48.729360    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804\": container with ID starting with 68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804 not found: ID does not exist" containerID="68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"
Jul 31 19:21:48.729468 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.729386    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804"} err="failed to get container status \"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804\": rpc error: code = NotFound desc = could not find container \"68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804\": container with ID starting with 68da76903c668d39055a6eb5eba828ca3cb63713f9b10d13de7a575cc6dad804 not found: ID does not exist"
Jul 31 19:21:48.739455 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.739363    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:21:48.742044 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.742022    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aaeacbf1-77d5fbd4df-k4k68"]
Jul 31 19:21:48.753479 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.753453    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:21:48.757939 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:48.757920    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aaeacbf1-58d54d5fcd-wkw5j"]
Jul 31 19:21:50.183347 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:50.183317    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147183f7-388f-4aed-8bcf-1a37698154a4" path="/var/lib/kubelet/pods/147183f7-388f-4aed-8bcf-1a37698154a4/volumes"
Jul 31 19:21:50.183725 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:50.183713    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cfd63e1-ea14-4524-b032-de269edeef12" path="/var/lib/kubelet/pods/2cfd63e1-ea14-4524-b032-de269edeef12/volumes"
Jul 31 19:21:50.184010 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:21:50.183998    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="526efde7-5d67-4858-8b32-9cbae4bba0f0" path="/var/lib/kubelet/pods/526efde7-5d67-4858-8b32-9cbae4bba0f0/volumes"
Jul 31 19:23:42.297217 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:42.297184    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:23:42.299515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:42.299481    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:23:50.974641 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.974606    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:23:50.975083 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975063    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfd63e1-ea14-4524-b032-de269edeef12" containerName="maas-api"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975088    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfd63e1-ea14-4524-b032-de269edeef12" containerName="maas-api"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975100    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb9098f-451f-4d3a-a44a-065357f43615" containerName="revoke-keys"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975109    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb9098f-451f-4d3a-a44a-065357f43615" containerName="revoke-keys"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975128    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147183f7-388f-4aed-8bcf-1a37698154a4" containerName="payload-processing"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975138    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="147183f7-388f-4aed-8bcf-1a37698154a4" containerName="payload-processing"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975152    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526efde7-5d67-4858-8b32-9cbae4bba0f0" containerName="payload-pre-processing"
Jul 31 19:23:50.975191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975161    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="526efde7-5d67-4858-8b32-9cbae4bba0f0" containerName="payload-pre-processing"
Jul 31 19:23:50.975473 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975249    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb9098f-451f-4d3a-a44a-065357f43615" containerName="revoke-keys"
Jul 31 19:23:50.975473 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975260    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfd63e1-ea14-4524-b032-de269edeef12" containerName="maas-api"
Jul 31 19:23:50.975473 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975273    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="147183f7-388f-4aed-8bcf-1a37698154a4" containerName="payload-processing"
Jul 31 19:23:50.975473 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.975284    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="526efde7-5d67-4858-8b32-9cbae4bba0f0" containerName="payload-pre-processing"
Jul 31 19:23:50.978478 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.978456    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:50.981557 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.981532    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-66ce83-serving-cert\""
Jul 31 19:23:50.981731 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.981710    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:23:50.981861 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.981651    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:23:50.982515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.982489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:23:50.992993 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:50.992964    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:23:51.057986 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.057944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.058166 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.058038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.058166 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.058100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dkv\" (UniqueName: \"kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.159379 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.159338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.159379 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.159383    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dkv\" (UniqueName: \"kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.159657 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.159515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.159717 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:23:51.159658    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-serving-cert: secret "maas-api-e2e-inf-a-66ce83-serving-cert" not found
Jul 31 19:23:51.159769 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:23:51.159733    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls podName:ba26cbdc-3306-4e66-906f-256cccb3fffa nodeName:}" failed. No retries permitted until 2026-07-31 19:23:51.659712762 +0000 UTC m=+2410.007294636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls") pod "maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" (UID: "ba26cbdc-3306-4e66-906f-256cccb3fffa") : secret "maas-api-e2e-inf-a-66ce83-serving-cert" not found
Jul 31 19:23:51.160167 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.160145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.171393 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.171364    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dkv\" (UniqueName: \"kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.663365 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.663327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.665963 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.665938    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") pod \"maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.836187 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.836151    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:23:51.839420 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.839404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:51.842193 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.842168    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-66ce83\""
Jul 31 19:23:51.853156 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.853136    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:23:51.872179 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.872155    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:23:51.875203 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.875183    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:51.877705 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.877683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-66ce83-dockercfg-2cz4m\""
Jul 31 19:23:51.888632 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.888611    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:51.891154 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.891134    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:23:51.965154 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.965127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2267b\" (UniqueName: \"kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:51.965305 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.965166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9prr\" (UniqueName: \"kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:51.965305 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.965198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:51.965377 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:51.965299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.048618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.042941    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:23:52.065966 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.065938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:52.066085 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.065979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.066085 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.066016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2267b\" (UniqueName: \"kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.066085 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.066040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9prr\" (UniqueName: \"kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:52.066617 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.066593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:52.066725 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.066708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.077936 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.077912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2267b\" (UniqueName: \"kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b\") pod \"payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.078684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.078662    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9prr\" (UniqueName: \"kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr\") pod \"payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:52.149832 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.149794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:52.184695 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.184670    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:52.306714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.306691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:23:52.309534 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:23:52.309501    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faebe63_f9f8_421f_88c4_763be7f0f73f.slice/crio-520a5840ba61ecd659863ab354e2c53c13ce338dfe30f92465037bcf8f7e6223 WatchSource:0}: Error finding container 520a5840ba61ecd659863ab354e2c53c13ce338dfe30f92465037bcf8f7e6223: Status 404 returned error can't find the container with id 520a5840ba61ecd659863ab354e2c53c13ce338dfe30f92465037bcf8f7e6223
Jul 31 19:23:52.311728 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.311641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" event={"ID":"ba26cbdc-3306-4e66-906f-256cccb3fffa","Type":"ContainerStarted","Data":"0572592305c9467440df7099f5179bfc2ec4f7383605a02bf8583217deb8fcd3"}
Jul 31 19:23:52.335655 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:52.335630    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:23:52.339881 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:23:52.339848    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0a79d0_61ab_4f6a_a2d3_bdccffc1a20f.slice/crio-15cd4a62fd273a7a8c522968647328df997a3dfc82908bef1f15f6a295292f7e WatchSource:0}: Error finding container 15cd4a62fd273a7a8c522968647328df997a3dfc82908bef1f15f6a295292f7e: Status 404 returned error can't find the container with id 15cd4a62fd273a7a8c522968647328df997a3dfc82908bef1f15f6a295292f7e
Jul 31 19:23:53.318349 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.318315    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" event={"ID":"5faebe63-f9f8-421f-88c4-763be7f0f73f","Type":"ContainerStarted","Data":"87241efbeba203873f825989c4d28c4c531697b14a2e6fcc8f91bdb8fe81b37b"}
Jul 31 19:23:53.318793 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.318366    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" event={"ID":"5faebe63-f9f8-421f-88c4-763be7f0f73f","Type":"ContainerStarted","Data":"520a5840ba61ecd659863ab354e2c53c13ce338dfe30f92465037bcf8f7e6223"}
Jul 31 19:23:53.318793 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.318385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:23:53.319847 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.319824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" event={"ID":"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f","Type":"ContainerStarted","Data":"9a1fa8408cf602b7c3e089df7d6ddf000effad78a48ebe1fffa454c3a78ce19e"}
Jul 31 19:23:53.319958 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.319872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" event={"ID":"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f","Type":"ContainerStarted","Data":"15cd4a62fd273a7a8c522968647328df997a3dfc82908bef1f15f6a295292f7e"}
Jul 31 19:23:53.319958 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.319901    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:23:53.321179 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.321159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" event={"ID":"ba26cbdc-3306-4e66-906f-256cccb3fffa","Type":"ContainerStarted","Data":"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"}
Jul 31 19:23:53.321249 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.321234    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:23:53.337939 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.337896    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" podStartSLOduration=2.337882398 podStartE2EDuration="2.337882398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:23:53.336959785 +0000 UTC m=+2411.684541669" watchObservedRunningTime="2026-07-31 19:23:53.337882398 +0000 UTC m=+2411.685464278"
Jul 31 19:23:53.355562 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.355517    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" podStartSLOduration=2.355503204 podStartE2EDuration="2.355503204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:23:53.354032729 +0000 UTC m=+2411.701614610" watchObservedRunningTime="2026-07-31 19:23:53.355503204 +0000 UTC m=+2411.703085084"
Jul 31 19:23:53.371631 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:53.371584    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" podStartSLOduration=2.843699587 podStartE2EDuration="3.371565105s" totalImagesPullingTime="527.865518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:23:53.370693424 +0000 UTC m=+2411.718275303" watchObservedRunningTime="2026-07-31 19:23:53.371565105 +0000 UTC m=+2411.719146986"
Jul 31 19:23:59.331789 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:23:59.331762    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:24:04.328910 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:04.328827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:24:04.329395 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:04.329130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:24:19.054205 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.054167    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:24:19.057549 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.057520    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.060208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.060181    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8aa450-serving-cert\""
Jul 31 19:24:19.070023 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.070002    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:24:19.087894 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.087866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.088005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.087895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6gv\" (UniqueName: \"kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.088005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.087927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.189406 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.189373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.189406 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.189409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6gv\" (UniqueName: \"kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.189637 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.189454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.189637 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:24:19.189555    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-serving-cert: secret "maas-api-e2e-inf-b-8aa450-serving-cert" not found
Jul 31 19:24:19.189728 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:24:19.189644    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls podName:209b635c-216d-407b-a153-c3ce2384e5a0 nodeName:}" failed. No retries permitted until 2026-07-31 19:24:19.689622931 +0000 UTC m=+2438.037204791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls") pod "maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" (UID: "209b635c-216d-407b-a153-c3ce2384e5a0") : secret "maas-api-e2e-inf-b-8aa450-serving-cert" not found
Jul 31 19:24:19.190081 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.190061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.199820 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.199798    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6gv\" (UniqueName: \"kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.694203 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.694172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.696807 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.696768    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:19.764428 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.764391    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:24:19.768379 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.768357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.771250 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.771225    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8aa450\""
Jul 31 19:24:19.771380 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.771289    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8aa450-dockercfg-k4dfv\""
Jul 31 19:24:19.778535 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.778508    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:24:19.794843 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.794814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hl6h\" (UniqueName: \"kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.794984 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.794897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.895739 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.895689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.895934 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.895820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hl6h\" (UniqueName: \"kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.896633 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.896614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.905336 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.905305    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hl6h\" (UniqueName: \"kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h\") pod \"payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:19.969763 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:19.969676    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:20.079829 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.079785    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:20.108831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.108802    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:24:20.110514 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:24:20.110489    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209b635c_216d_407b_a153_c3ce2384e5a0.slice/crio-9bf2531f58e9117a9fd65a2d677e21279d58ccfbbe468b3c5ff97eb06e00fb19 WatchSource:0}: Error finding container 9bf2531f58e9117a9fd65a2d677e21279d58ccfbbe468b3c5ff97eb06e00fb19: Status 404 returned error can't find the container with id 9bf2531f58e9117a9fd65a2d677e21279d58ccfbbe468b3c5ff97eb06e00fb19
Jul 31 19:24:20.223430 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.223401    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:24:20.225168 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:24:20.225143    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4701c5ff_fe60_4cae_b737_478bfb59b28a.slice/crio-3d0f41820ef89bf83d54751e4e6809b697a52a76764ca3c6e68fd8faa6268839 WatchSource:0}: Error finding container 3d0f41820ef89bf83d54751e4e6809b697a52a76764ca3c6e68fd8faa6268839: Status 404 returned error can't find the container with id 3d0f41820ef89bf83d54751e4e6809b697a52a76764ca3c6e68fd8faa6268839
Jul 31 19:24:20.456233 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.456207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" event={"ID":"209b635c-216d-407b-a153-c3ce2384e5a0","Type":"ContainerStarted","Data":"9bf2531f58e9117a9fd65a2d677e21279d58ccfbbe468b3c5ff97eb06e00fb19"}
Jul 31 19:24:20.457739 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.457717    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" event={"ID":"4701c5ff-fe60-4cae-b737-478bfb59b28a","Type":"ContainerStarted","Data":"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"}
Jul 31 19:24:20.457860 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.457757    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" event={"ID":"4701c5ff-fe60-4cae-b737-478bfb59b28a","Type":"ContainerStarted","Data":"3d0f41820ef89bf83d54751e4e6809b697a52a76764ca3c6e68fd8faa6268839"}
Jul 31 19:24:20.457860 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.457848    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:20.476051 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:20.475982    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" podStartSLOduration=1.475968212 podStartE2EDuration="1.475968212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:24:20.474637212 +0000 UTC m=+2438.822219092" watchObservedRunningTime="2026-07-31 19:24:20.475968212 +0000 UTC m=+2438.823550084"
Jul 31 19:24:21.464242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:21.464205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" event={"ID":"209b635c-216d-407b-a153-c3ce2384e5a0","Type":"ContainerStarted","Data":"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"}
Jul 31 19:24:21.464693 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:21.464571    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:21.483261 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:21.483215    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" podStartSLOduration=2.153191129 podStartE2EDuration="2.483203104s" totalImagesPullingTime="330.011975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:24:21.481686797 +0000 UTC m=+2439.829268676" watchObservedRunningTime="2026-07-31 19:24:21.483203104 +0000 UTC m=+2439.830784983"
Jul 31 19:24:27.476673 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:27.476642    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:24:31.466534 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:31.466505    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:24:42.441742 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:24:42.441705    2557 scope.go:117] "RemoveContainer" containerID="0a75f2089a1b256d7a8d6519bbb756dadc0dcd38d07844b3d9b3b69c158d6cc7"
Jul 31 19:25:58.219115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.219035    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"]
Jul 31 19:25:58.222409 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.222391    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:25:58.225317 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.225285    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:25:58.231596 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.231569    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"]
Jul 31 19:25:58.234765 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.234740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f\" (UID: \"0770ad51-4d6c-4fc2-be62-006f2beb1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:25:58.335576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.335540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f\" (UID: \"0770ad51-4d6c-4fc2-be62-006f2beb1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:25:58.336151 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.336132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f\" (UID: \"0770ad51-4d6c-4fc2-be62-006f2beb1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:25:58.534371 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.534276    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:25:58.668075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.668051    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"]
Jul 31 19:25:58.670038 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:25:58.670011    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0770ad51_4d6c_4fc2_be62_006f2beb1c4e.slice/crio-c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6 WatchSource:0}: Error finding container c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6: Status 404 returned error can't find the container with id c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6
Jul 31 19:25:58.947782 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:58.947746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f" event={"ID":"0770ad51-4d6c-4fc2-be62-006f2beb1c4e","Type":"ContainerStarted","Data":"c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6"}
Jul 31 19:25:59.954677 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:59.954646    2557 generic.go:356] "Generic (PLEG): container finished" podID="0770ad51-4d6c-4fc2-be62-006f2beb1c4e" containerID="173ed1fc38278a11f87a51a8dec702e89bf7fa9de6427eaa6201bfc2ffbe4134" exitCode=0
Jul 31 19:25:59.955090 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:25:59.954737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f" event={"ID":"0770ad51-4d6c-4fc2-be62-006f2beb1c4e","Type":"ContainerDied","Data":"173ed1fc38278a11f87a51a8dec702e89bf7fa9de6427eaa6201bfc2ffbe4134"}
Jul 31 19:26:01.087772 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.087747    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:26:01.154155 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.154119    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca\") pod \"0770ad51-4d6c-4fc2-be62-006f2beb1c4e\" (UID: \"0770ad51-4d6c-4fc2-be62-006f2beb1c4e\") "
Jul 31 19:26:01.154526 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.154506    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0770ad51-4d6c-4fc2-be62-006f2beb1c4e" (UID: "0770ad51-4d6c-4fc2-be62-006f2beb1c4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:01.255178 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.255087    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0770ad51-4d6c-4fc2-be62-006f2beb1c4e-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:01.966496 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.966463    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"
Jul 31 19:26:01.966496 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.966468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f" event={"ID":"0770ad51-4d6c-4fc2-be62-006f2beb1c4e","Type":"ContainerDied","Data":"c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6"}
Jul 31 19:26:01.966714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:01.966513    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f4909ebe2e36cefbf24baee186a55818571bb238e1cc1fa2d0349f0df6b3a6"
Jul 31 19:26:08.791221 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.791188    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:26:08.792079 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.791410    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" podUID="ba26cbdc-3306-4e66-906f-256cccb3fffa" containerName="maas-api" containerID="cri-o://05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0" gracePeriod=30
Jul 31 19:26:08.852736 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.852700    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:26:08.852963 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.852939    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" podUID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" containerName="payload-processing" containerID="cri-o://9a1fa8408cf602b7c3e089df7d6ddf000effad78a48ebe1fffa454c3a78ce19e" gracePeriod=30
Jul 31 19:26:08.861601 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.861571    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:26:08.861819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:08.861794    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" podUID="5faebe63-f9f8-421f-88c4-763be7f0f73f" containerName="payload-pre-processing" containerID="cri-o://87241efbeba203873f825989c4d28c4c531697b14a2e6fcc8f91bdb8fe81b37b" gracePeriod=30
Jul 31 19:26:09.002828 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.002773    2557 generic.go:356] "Generic (PLEG): container finished" podID="5faebe63-f9f8-421f-88c4-763be7f0f73f" containerID="87241efbeba203873f825989c4d28c4c531697b14a2e6fcc8f91bdb8fe81b37b" exitCode=0
Jul 31 19:26:09.003049 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.002903    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" event={"ID":"5faebe63-f9f8-421f-88c4-763be7f0f73f","Type":"ContainerDied","Data":"87241efbeba203873f825989c4d28c4c531697b14a2e6fcc8f91bdb8fe81b37b"}
Jul 31 19:26:09.004852 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.004803    2557 generic.go:356] "Generic (PLEG): container finished" podID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" containerID="9a1fa8408cf602b7c3e089df7d6ddf000effad78a48ebe1fffa454c3a78ce19e" exitCode=0
Jul 31 19:26:09.004997 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.004862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" event={"ID":"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f","Type":"ContainerDied","Data":"9a1fa8408cf602b7c3e089df7d6ddf000effad78a48ebe1fffa454c3a78ce19e"}
Jul 31 19:26:09.025098 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.025065    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"]
Jul 31 19:26:09.028864 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.028365    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-66ce83-9sw9f"]
Jul 31 19:26:09.153639 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.153604    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:26:09.156810 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.156786    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:26:09.209911 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.209880    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9prr\" (UniqueName: \"kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr\") pod \"5faebe63-f9f8-421f-88c4-763be7f0f73f\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") "
Jul 31 19:26:09.210109 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.209945    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume\") pod \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") "
Jul 31 19:26:09.210109 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.209978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2267b\" (UniqueName: \"kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b\") pod \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\" (UID: \"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f\") "
Jul 31 19:26:09.210109 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.210015    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume\") pod \"5faebe63-f9f8-421f-88c4-763be7f0f73f\" (UID: \"5faebe63-f9f8-421f-88c4-763be7f0f73f\") "
Jul 31 19:26:09.210560 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.210528    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" (UID: "7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:09.210671 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.210595    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5faebe63-f9f8-421f-88c4-763be7f0f73f" (UID: "5faebe63-f9f8-421f-88c4-763be7f0f73f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:09.213370 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.213336    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b" (OuterVolumeSpecName: "kube-api-access-2267b") pod "7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" (UID: "7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f"). InnerVolumeSpecName "kube-api-access-2267b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:26:09.213499 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.213460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr" (OuterVolumeSpecName: "kube-api-access-v9prr") pod "5faebe63-f9f8-421f-88c4-763be7f0f73f" (UID: "5faebe63-f9f8-421f-88c4-763be7f0f73f"). InnerVolumeSpecName "kube-api-access-v9prr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:26:09.237814 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.237788    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:26:09.311502 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311435    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8dkv\" (UniqueName: \"kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv\") pod \"ba26cbdc-3306-4e66-906f-256cccb3fffa\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") "
Jul 31 19:26:09.311502 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311505    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca\") pod \"ba26cbdc-3306-4e66-906f-256cccb3fffa\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") "
Jul 31 19:26:09.311767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311534    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") pod \"ba26cbdc-3306-4e66-906f-256cccb3fffa\" (UID: \"ba26cbdc-3306-4e66-906f-256cccb3fffa\") "
Jul 31 19:26:09.311767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311663    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9prr\" (UniqueName: \"kubernetes.io/projected/5faebe63-f9f8-421f-88c4-763be7f0f73f-kube-api-access-v9prr\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.311767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311679    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.311767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311692    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2267b\" (UniqueName: \"kubernetes.io/projected/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f-kube-api-access-2267b\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.311767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311702    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faebe63-f9f8-421f-88c4-763be7f0f73f-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.311990 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.311852    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba26cbdc-3306-4e66-906f-256cccb3fffa" (UID: "ba26cbdc-3306-4e66-906f-256cccb3fffa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:09.313798 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.313748    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba26cbdc-3306-4e66-906f-256cccb3fffa" (UID: "ba26cbdc-3306-4e66-906f-256cccb3fffa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:26:09.313910 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.313885    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv" (OuterVolumeSpecName: "kube-api-access-z8dkv") pod "ba26cbdc-3306-4e66-906f-256cccb3fffa" (UID: "ba26cbdc-3306-4e66-906f-256cccb3fffa"). InnerVolumeSpecName "kube-api-access-z8dkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:26:09.412957 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.412921    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dkv\" (UniqueName: \"kubernetes.io/projected/ba26cbdc-3306-4e66-906f-256cccb3fffa-kube-api-access-z8dkv\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.412957 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.412957    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba26cbdc-3306-4e66-906f-256cccb3fffa-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:09.413169 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:09.412971    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba26cbdc-3306-4e66-906f-256cccb3fffa-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:10.011935 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.011896    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba26cbdc-3306-4e66-906f-256cccb3fffa" containerID="05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0" exitCode=0
Jul 31 19:26:10.012515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.011954    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"
Jul 31 19:26:10.012515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.011990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" event={"ID":"ba26cbdc-3306-4e66-906f-256cccb3fffa","Type":"ContainerDied","Data":"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"}
Jul 31 19:26:10.012515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.012055    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf" event={"ID":"ba26cbdc-3306-4e66-906f-256cccb3fffa","Type":"ContainerDied","Data":"0572592305c9467440df7099f5179bfc2ec4f7383605a02bf8583217deb8fcd3"}
Jul 31 19:26:10.012515 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.012076    2557 scope.go:117] "RemoveContainer" containerID="05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"
Jul 31 19:26:10.013514 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.013484    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"
Jul 31 19:26:10.013514 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.013497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j" event={"ID":"5faebe63-f9f8-421f-88c4-763be7f0f73f","Type":"ContainerDied","Data":"520a5840ba61ecd659863ab354e2c53c13ce338dfe30f92465037bcf8f7e6223"}
Jul 31 19:26:10.015131 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.015108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv" event={"ID":"7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f","Type":"ContainerDied","Data":"15cd4a62fd273a7a8c522968647328df997a3dfc82908bef1f15f6a295292f7e"}
Jul 31 19:26:10.015131 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.015130    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"
Jul 31 19:26:10.025510 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.025473    2557 scope.go:117] "RemoveContainer" containerID="05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"
Jul 31 19:26:10.025908 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:26:10.025866    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0\": container with ID starting with 05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0 not found: ID does not exist" containerID="05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"
Jul 31 19:26:10.026017 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.025920    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0"} err="failed to get container status \"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0\": rpc error: code = NotFound desc = could not find container \"05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0\": container with ID starting with 05e0a720653ffc2a88dae4ec6ae0898ee596106e2b58277f27e856d38463beb0 not found: ID does not exist"
Jul 31 19:26:10.026017 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.025945    2557 scope.go:117] "RemoveContainer" containerID="87241efbeba203873f825989c4d28c4c531697b14a2e6fcc8f91bdb8fe81b37b"
Jul 31 19:26:10.035489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.035471    2557 scope.go:117] "RemoveContainer" containerID="9a1fa8408cf602b7c3e089df7d6ddf000effad78a48ebe1fffa454c3a78ce19e"
Jul 31 19:26:10.048383 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.048359    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:26:10.052769 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.052743    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-66ce83-665576d6bb-97pxf"]
Jul 31 19:26:10.064876 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.064855    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:26:10.069385 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.069365    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-66ce83-6ff79dc8d-chf9j"]
Jul 31 19:26:10.081291 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.081268    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:26:10.085692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.085665    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-66ce83-775cfffd84-ljqfv"]
Jul 31 19:26:10.183367 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.183283    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0770ad51-4d6c-4fc2-be62-006f2beb1c4e" path="/var/lib/kubelet/pods/0770ad51-4d6c-4fc2-be62-006f2beb1c4e/volumes"
Jul 31 19:26:10.183755 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.183735    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5faebe63-f9f8-421f-88c4-763be7f0f73f" path="/var/lib/kubelet/pods/5faebe63-f9f8-421f-88c4-763be7f0f73f/volumes"
Jul 31 19:26:10.184106 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.184089    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" path="/var/lib/kubelet/pods/7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f/volumes"
Jul 31 19:26:10.184494 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:10.184477    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba26cbdc-3306-4e66-906f-256cccb3fffa" path="/var/lib/kubelet/pods/ba26cbdc-3306-4e66-906f-256cccb3fffa/volumes"
Jul 31 19:26:16.704731 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.704692    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"]
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705006    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" containerName="payload-processing"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705016    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" containerName="payload-processing"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705032    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0770ad51-4d6c-4fc2-be62-006f2beb1c4e" containerName="revoke-keys"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705039    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0770ad51-4d6c-4fc2-be62-006f2beb1c4e" containerName="revoke-keys"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705049    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba26cbdc-3306-4e66-906f-256cccb3fffa" containerName="maas-api"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705055    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba26cbdc-3306-4e66-906f-256cccb3fffa" containerName="maas-api"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faebe63-f9f8-421f-88c4-763be7f0f73f" containerName="payload-pre-processing"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faebe63-f9f8-421f-88c4-763be7f0f73f" containerName="payload-pre-processing"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705116    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0770ad51-4d6c-4fc2-be62-006f2beb1c4e" containerName="revoke-keys"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705123    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba26cbdc-3306-4e66-906f-256cccb3fffa" containerName="maas-api"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705130    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5faebe63-f9f8-421f-88c4-763be7f0f73f" containerName="payload-pre-processing"
Jul 31 19:26:16.705138 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.705136    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0a79d0-61ab-4f6a-a2d3-bdccffc1a20f" containerName="payload-processing"
Jul 31 19:26:16.709625 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.709606    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:16.714160 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.714142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:26:16.720124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.720099    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"]
Jul 31 19:26:16.774089 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.774055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6\" (UID: \"f67f1e13-da1e-4385-b33e-c79fce285b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:16.875391 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.875355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6\" (UID: \"f67f1e13-da1e-4385-b33e-c79fce285b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:16.875994 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:16.875974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6\" (UID: \"f67f1e13-da1e-4385-b33e-c79fce285b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:17.020224 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:17.020126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:17.154635 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:17.154609    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"]
Jul 31 19:26:17.156253 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:26:17.156218    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67f1e13_da1e_4385_b33e_c79fce285b43.slice/crio-387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513 WatchSource:0}: Error finding container 387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513: Status 404 returned error can't find the container with id 387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513
Jul 31 19:26:18.061993 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:18.061953    2557 generic.go:356] "Generic (PLEG): container finished" podID="f67f1e13-da1e-4385-b33e-c79fce285b43" containerID="d29ee568f9ac1d775a78bb034ac0f1ecf5f0570ffb185e7d7925da36024bcc53" exitCode=0
Jul 31 19:26:18.062385 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:18.062042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6" event={"ID":"f67f1e13-da1e-4385-b33e-c79fce285b43","Type":"ContainerDied","Data":"d29ee568f9ac1d775a78bb034ac0f1ecf5f0570ffb185e7d7925da36024bcc53"}
Jul 31 19:26:18.062385 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:18.062094    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6" event={"ID":"f67f1e13-da1e-4385-b33e-c79fce285b43","Type":"ContainerStarted","Data":"387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513"}
Jul 31 19:26:19.193686 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:19.193663    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:19.295391 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:19.295351    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca\") pod \"f67f1e13-da1e-4385-b33e-c79fce285b43\" (UID: \"f67f1e13-da1e-4385-b33e-c79fce285b43\") "
Jul 31 19:26:19.295747 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:19.295719    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f67f1e13-da1e-4385-b33e-c79fce285b43" (UID: "f67f1e13-da1e-4385-b33e-c79fce285b43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:19.396762 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:19.396722    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67f1e13-da1e-4385-b33e-c79fce285b43-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:20.072554 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:20.072518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6" event={"ID":"f67f1e13-da1e-4385-b33e-c79fce285b43","Type":"ContainerDied","Data":"387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513"}
Jul 31 19:26:20.072554 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:20.072538    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"
Jul 31 19:26:20.072772 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:20.072584    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387dea3376f578f4e021ce188089450fe65990ba6caba39f6a55968f2519d513"
Jul 31 19:26:27.399119 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.399082    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:26:27.399555 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.399392    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" containerName="maas-api" containerID="cri-o://050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0" gracePeriod=30
Jul 31 19:26:27.456583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.456548    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:26:27.456789 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.456764    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" podUID="4701c5ff-fe60-4cae-b737-478bfb59b28a" containerName="payload-pre-processing" containerID="cri-o://3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac" gracePeriod=30
Jul 31 19:26:27.470538 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.470506    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": dial tcp 10.132.0.55:8443: connect: connection refused"
Jul 31 19:26:27.597644 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.597606    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"]
Jul 31 19:26:27.601198 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.601106    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8aa450-k78v6"]
Jul 31 19:26:27.711530 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.711507    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:26:27.727518 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.727498    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:26:27.771050 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771017    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j6gv\" (UniqueName: \"kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv\") pod \"209b635c-216d-407b-a153-c3ce2384e5a0\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") "
Jul 31 19:26:27.771237 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771088    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") pod \"209b635c-216d-407b-a153-c3ce2384e5a0\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") "
Jul 31 19:26:27.771237 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume\") pod \"4701c5ff-fe60-4cae-b737-478bfb59b28a\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") "
Jul 31 19:26:27.771361 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771248    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hl6h\" (UniqueName: \"kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h\") pod \"4701c5ff-fe60-4cae-b737-478bfb59b28a\" (UID: \"4701c5ff-fe60-4cae-b737-478bfb59b28a\") "
Jul 31 19:26:27.771361 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771293    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca\") pod \"209b635c-216d-407b-a153-c3ce2384e5a0\" (UID: \"209b635c-216d-407b-a153-c3ce2384e5a0\") "
Jul 31 19:26:27.771682 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4701c5ff-fe60-4cae-b737-478bfb59b28a" (UID: "4701c5ff-fe60-4cae-b737-478bfb59b28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:27.771784 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.771706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "209b635c-216d-407b-a153-c3ce2384e5a0" (UID: "209b635c-216d-407b-a153-c3ce2384e5a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:26:27.773417 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.773395    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "209b635c-216d-407b-a153-c3ce2384e5a0" (UID: "209b635c-216d-407b-a153-c3ce2384e5a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:26:27.773417 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.773398    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv" (OuterVolumeSpecName: "kube-api-access-7j6gv") pod "209b635c-216d-407b-a153-c3ce2384e5a0" (UID: "209b635c-216d-407b-a153-c3ce2384e5a0"). InnerVolumeSpecName "kube-api-access-7j6gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:26:27.773417 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.773393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h" (OuterVolumeSpecName: "kube-api-access-6hl6h") pod "4701c5ff-fe60-4cae-b737-478bfb59b28a" (UID: "4701c5ff-fe60-4cae-b737-478bfb59b28a"). InnerVolumeSpecName "kube-api-access-6hl6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:26:27.872576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.872545    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/209b635c-216d-407b-a153-c3ce2384e5a0-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:27.872576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.872569    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4701c5ff-fe60-4cae-b737-478bfb59b28a-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:27.872576 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.872578    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hl6h\" (UniqueName: \"kubernetes.io/projected/4701c5ff-fe60-4cae-b737-478bfb59b28a-kube-api-access-6hl6h\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:27.872798 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.872587    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b635c-216d-407b-a153-c3ce2384e5a0-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:27.872798 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:27.872595    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6gv\" (UniqueName: \"kubernetes.io/projected/209b635c-216d-407b-a153-c3ce2384e5a0-kube-api-access-7j6gv\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:26:28.113599 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.113565    2557 generic.go:356] "Generic (PLEG): container finished" podID="209b635c-216d-407b-a153-c3ce2384e5a0" containerID="050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0" exitCode=0
Jul 31 19:26:28.113838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.113611    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"
Jul 31 19:26:28.113838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.113660    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" event={"ID":"209b635c-216d-407b-a153-c3ce2384e5a0","Type":"ContainerDied","Data":"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"}
Jul 31 19:26:28.113838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.113716    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df" event={"ID":"209b635c-216d-407b-a153-c3ce2384e5a0","Type":"ContainerDied","Data":"9bf2531f58e9117a9fd65a2d677e21279d58ccfbbe468b3c5ff97eb06e00fb19"}
Jul 31 19:26:28.113838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.113731    2557 scope.go:117] "RemoveContainer" containerID="050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"
Jul 31 19:26:28.115203 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.115176    2557 generic.go:356] "Generic (PLEG): container finished" podID="4701c5ff-fe60-4cae-b737-478bfb59b28a" containerID="3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac" exitCode=0
Jul 31 19:26:28.115317 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.115228    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"
Jul 31 19:26:28.115317 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.115232    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" event={"ID":"4701c5ff-fe60-4cae-b737-478bfb59b28a","Type":"ContainerDied","Data":"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"}
Jul 31 19:26:28.115317 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.115265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn" event={"ID":"4701c5ff-fe60-4cae-b737-478bfb59b28a","Type":"ContainerDied","Data":"3d0f41820ef89bf83d54751e4e6809b697a52a76764ca3c6e68fd8faa6268839"}
Jul 31 19:26:28.123767 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.123748    2557 scope.go:117] "RemoveContainer" containerID="050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"
Jul 31 19:26:28.124038 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:26:28.124022    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0\": container with ID starting with 050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0 not found: ID does not exist" containerID="050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"
Jul 31 19:26:28.124096 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.124046    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0"} err="failed to get container status \"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0\": rpc error: code = NotFound desc = could not find container \"050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0\": container with ID starting with 050e357548040a399c7c9d1cef2e9d5948116fc7abb963024e59b7b1a4910ec0 not found: ID does not exist"
Jul 31 19:26:28.124096 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.124061    2557 scope.go:117] "RemoveContainer" containerID="3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"
Jul 31 19:26:28.131687 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.131669    2557 scope.go:117] "RemoveContainer" containerID="3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"
Jul 31 19:26:28.131947 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:26:28.131927    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac\": container with ID starting with 3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac not found: ID does not exist" containerID="3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"
Jul 31 19:26:28.132032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.131951    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac"} err="failed to get container status \"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac\": rpc error: code = NotFound desc = could not find container \"3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac\": container with ID starting with 3c1f2f4a461ed7a17cb615001ba436a13fe33967e928f63e1c07eed3f49a04ac not found: ID does not exist"
Jul 31 19:26:28.137856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.137834    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:26:28.144102 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.144082    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8aa450-8595cb6c-vg7df"]
Jul 31 19:26:28.156558 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.156513    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:26:28.158602 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.158584    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8aa450-85664b979b-79pvn"]
Jul 31 19:26:28.182678 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.182656    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" path="/var/lib/kubelet/pods/209b635c-216d-407b-a153-c3ce2384e5a0/volumes"
Jul 31 19:26:28.183007 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.182994    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4701c5ff-fe60-4cae-b737-478bfb59b28a" path="/var/lib/kubelet/pods/4701c5ff-fe60-4cae-b737-478bfb59b28a/volumes"
Jul 31 19:26:28.183277 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:28.183266    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67f1e13-da1e-4385-b33e-c79fce285b43" path="/var/lib/kubelet/pods/f67f1e13-da1e-4385-b33e-c79fce285b43/volumes"
Jul 31 19:26:39.692110 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692072    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692501    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67f1e13-da1e-4385-b33e-c79fce285b43" containerName="revoke-keys"
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692517    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67f1e13-da1e-4385-b33e-c79fce285b43" containerName="revoke-keys"
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692532    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" containerName="maas-api"
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692541    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" containerName="maas-api"
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692573    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4701c5ff-fe60-4cae-b737-478bfb59b28a" containerName="payload-pre-processing"
Jul 31 19:26:39.692658 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692583    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4701c5ff-fe60-4cae-b737-478bfb59b28a" containerName="payload-pre-processing"
Jul 31 19:26:39.692955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692663    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4701c5ff-fe60-4cae-b737-478bfb59b28a" containerName="payload-pre-processing"
Jul 31 19:26:39.692955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692679    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="209b635c-216d-407b-a153-c3ce2384e5a0" containerName="maas-api"
Jul 31 19:26:39.692955 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.692688    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67f1e13-da1e-4385-b33e-c79fce285b43" containerName="revoke-keys"
Jul 31 19:26:39.697752 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.697733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.700353 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.700332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:26:39.700474 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.700456    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-968afa-serving-cert\""
Jul 31 19:26:39.702048 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.702027    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:26:39.702146 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.702027    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:26:39.706069 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.706032    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:26:39.775783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.775750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.776055 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.775807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwkx\" (UniqueName: \"kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.776055 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.775960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.877470 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.877400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwkx\" (UniqueName: \"kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.877684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.877497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.877684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.877519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.877684 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:26:39.877645    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-serving-cert: secret "maas-api-e2e-api-a-968afa-serving-cert" not found
Jul 31 19:26:39.877848 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:26:39.877725    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls podName:f7e32cd6-1b03-49a9-b2e1-6d0aef902e49 nodeName:}" failed. No retries permitted until 2026-07-31 19:26:40.377707301 +0000 UTC m=+2578.725289161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls") pod "maas-api-e2e-api-a-968afa-57b7589df9-km5h2" (UID: "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49") : secret "maas-api-e2e-api-a-968afa-serving-cert" not found
Jul 31 19:26:39.878069 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.878051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:39.888324 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:39.888299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwkx\" (UniqueName: \"kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:40.381242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.381204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:40.383894 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.383861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") pod \"maas-api-e2e-api-a-968afa-57b7589df9-km5h2\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:40.429763 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.429719    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:26:40.433242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.433225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.437075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.437051    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-968afa\""
Jul 31 19:26:40.437261 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.437124    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-968afa-dockercfg-b82kb\""
Jul 31 19:26:40.439339 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.439314    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:26:40.582623 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.582586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzm6r\" (UniqueName: \"kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.582816 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.582634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.610587 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.610550    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:40.683860 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.683828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzm6r\" (UniqueName: \"kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.684020 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.683896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.684610 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.684593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.693366 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.693344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzm6r\" (UniqueName: \"kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r\") pod \"payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") " pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.742948 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.742914    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:26:40.744739 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:26:40.744718    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e32cd6_1b03_49a9_b2e1_6d0aef902e49.slice/crio-bf7443a139ccf2077e7448260741900a59508479c4ff7d6eaefe4bd330798bd1 WatchSource:0}: Error finding container bf7443a139ccf2077e7448260741900a59508479c4ff7d6eaefe4bd330798bd1: Status 404 returned error can't find the container with id bf7443a139ccf2077e7448260741900a59508479c4ff7d6eaefe4bd330798bd1
Jul 31 19:26:40.745372 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.745357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:40.746085 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.746067    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 19:26:40.883734 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:40.883682    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:26:40.886368 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:26:40.886342    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4533d9_89ee_44f7_8523_798a594e21a6.slice/crio-6a9d59b45c1f50a095cfecdb2b8c676a69eb6f6c93d6cb8845d5597989b9922e WatchSource:0}: Error finding container 6a9d59b45c1f50a095cfecdb2b8c676a69eb6f6c93d6cb8845d5597989b9922e: Status 404 returned error can't find the container with id 6a9d59b45c1f50a095cfecdb2b8c676a69eb6f6c93d6cb8845d5597989b9922e
Jul 31 19:26:41.180382 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:41.180286    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" event={"ID":"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49","Type":"ContainerStarted","Data":"bf7443a139ccf2077e7448260741900a59508479c4ff7d6eaefe4bd330798bd1"}
Jul 31 19:26:41.181894 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:41.181865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" event={"ID":"6c4533d9-89ee-44f7-8523-798a594e21a6","Type":"ContainerStarted","Data":"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"}
Jul 31 19:26:41.182028 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:41.181914    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" event={"ID":"6c4533d9-89ee-44f7-8523-798a594e21a6","Type":"ContainerStarted","Data":"6a9d59b45c1f50a095cfecdb2b8c676a69eb6f6c93d6cb8845d5597989b9922e"}
Jul 31 19:26:41.182028 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:41.181944    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:26:41.202783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:41.202729    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" podStartSLOduration=1.202708906 podStartE2EDuration="1.202708906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:26:41.201503471 +0000 UTC m=+2579.549085351" watchObservedRunningTime="2026-07-31 19:26:41.202708906 +0000 UTC m=+2579.550290787"
Jul 31 19:26:42.187362 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:42.187327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" event={"ID":"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49","Type":"ContainerStarted","Data":"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"}
Jul 31 19:26:42.187783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:42.187611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:42.207459 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:42.207399    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" podStartSLOduration=2.613427062 podStartE2EDuration="3.207385459s" totalImagesPullingTime="593.958397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:26:42.20574684 +0000 UTC m=+2580.553328725" watchObservedRunningTime="2026-07-31 19:26:42.207385459 +0000 UTC m=+2580.554967338"
Jul 31 19:26:48.198874 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:48.198844    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:26:52.190024 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:26:52.189994    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:27:08.065470 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.065366    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:27:08.069210 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.069187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.072140 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.072118    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f3dafe-serving-cert\""
Jul 31 19:27:08.079860 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.079837    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:27:08.097454 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.097413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.097594 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.097501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.097594 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.097567    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvpc\" (UniqueName: \"kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.198852 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.198820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvpc\" (UniqueName: \"kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.199059 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.198899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.199059 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.198974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.199188 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:27:08.199086    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-serving-cert: secret "maas-api-e2e-api-b-f3dafe-serving-cert" not found
Jul 31 19:27:08.199188 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:27:08.199176    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls podName:a7364346-6497-46d7-a35a-2321b7455977 nodeName:}" failed. No retries permitted until 2026-07-31 19:27:08.69915406 +0000 UTC m=+2607.046735928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls") pod "maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" (UID: "a7364346-6497-46d7-a35a-2321b7455977") : secret "maas-api-e2e-api-b-f3dafe-serving-cert" not found
Jul 31 19:27:08.199594 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.199578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.209928 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.209893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvpc\" (UniqueName: \"kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.701898 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.701864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.704512 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.704480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") pod \"maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:08.755118 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.755081    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:27:08.758423 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.758407    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.761242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.761220    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f3dafe-dockercfg-95rwq\""
Jul 31 19:27:08.761375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.761292    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f3dafe\""
Jul 31 19:27:08.768709 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.768689    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:27:08.803058 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.803023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mfj\" (UniqueName: \"kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.803242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.803128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.815621 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.815590    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:27:08.818831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.818814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:08.828510 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.828484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:27:08.904297 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.904260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mfj\" (UniqueName: \"kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.904503 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.904302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkd5\" (UniqueName: \"kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:08.904503 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.904371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.904503 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.904435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:08.905012 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.904993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.913457 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.913428    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mfj\" (UniqueName: \"kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj\") pod \"payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:08.982396 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:08.982307    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:09.005347 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.005319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkd5\" (UniqueName: \"kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:09.005498 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.005402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:09.006023 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.006001    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:09.015275 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.015243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkd5\" (UniqueName: \"kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5\") pod \"payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:09.069796 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.069763    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:09.118394 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.118369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:27:09.120218 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:27:09.120186    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7364346_6497_46d7_a35a_2321b7455977.slice/crio-b58b9eb57d0a93642164e4a460a97194a38c03a5d2445a01549464ea522768fe WatchSource:0}: Error finding container b58b9eb57d0a93642164e4a460a97194a38c03a5d2445a01549464ea522768fe: Status 404 returned error can't find the container with id b58b9eb57d0a93642164e4a460a97194a38c03a5d2445a01549464ea522768fe
Jul 31 19:27:09.131494 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.131424    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:09.217650 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.217619    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:27:09.221044 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:27:09.221014    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd8091_bfcc_434b_a44c_fb826e7f01c5.slice/crio-be60b5d4028664e49ded35bd302897ee9f7caa94c12aacc1218a927354604e39 WatchSource:0}: Error finding container be60b5d4028664e49ded35bd302897ee9f7caa94c12aacc1218a927354604e39: Status 404 returned error can't find the container with id be60b5d4028664e49ded35bd302897ee9f7caa94c12aacc1218a927354604e39
Jul 31 19:27:09.281636 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.281605    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:27:09.287485 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:27:09.287461    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7309ce6_16aa_412a_81b4_ec82687bf497.slice/crio-a4eb51b02fc61eb4f0b6f358ac5217ced2e53fd008c69a2a7c57e7f45b982eed WatchSource:0}: Error finding container a4eb51b02fc61eb4f0b6f358ac5217ced2e53fd008c69a2a7c57e7f45b982eed: Status 404 returned error can't find the container with id a4eb51b02fc61eb4f0b6f358ac5217ced2e53fd008c69a2a7c57e7f45b982eed
Jul 31 19:27:09.327203 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.327171    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" event={"ID":"d7309ce6-16aa-412a-81b4-ec82687bf497","Type":"ContainerStarted","Data":"a4eb51b02fc61eb4f0b6f358ac5217ced2e53fd008c69a2a7c57e7f45b982eed"}
Jul 31 19:27:09.328595 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.328557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" event={"ID":"a7364346-6497-46d7-a35a-2321b7455977","Type":"ContainerStarted","Data":"b58b9eb57d0a93642164e4a460a97194a38c03a5d2445a01549464ea522768fe"}
Jul 31 19:27:09.330208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.330185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" event={"ID":"6ebd8091-bfcc-434b-a44c-fb826e7f01c5","Type":"ContainerStarted","Data":"08b84a455fc4609983d79ae17363c0679f317daad455667d735ac1f063a6710b"}
Jul 31 19:27:09.330344 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.330225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" event={"ID":"6ebd8091-bfcc-434b-a44c-fb826e7f01c5","Type":"ContainerStarted","Data":"be60b5d4028664e49ded35bd302897ee9f7caa94c12aacc1218a927354604e39"}
Jul 31 19:27:09.330344 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.330262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:09.348811 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:09.348773    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" podStartSLOduration=1.348758956 podStartE2EDuration="1.348758956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:27:09.346608397 +0000 UTC m=+2607.694190279" watchObservedRunningTime="2026-07-31 19:27:09.348758956 +0000 UTC m=+2607.696340881"
Jul 31 19:27:10.336834 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:10.336800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" event={"ID":"a7364346-6497-46d7-a35a-2321b7455977","Type":"ContainerStarted","Data":"2cc8344d477cb1117fc43e2c2c9adcdaf3ed643ad5294843bacbcd94ba3f6d2e"}
Jul 31 19:27:10.337332 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:10.336882    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:10.338474 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:10.338435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" event={"ID":"d7309ce6-16aa-412a-81b4-ec82687bf497","Type":"ContainerStarted","Data":"413e4ae3d2a73dff1fe89db2972c5b659a41b86f553e307ccde490b07e67108e"}
Jul 31 19:27:10.356456 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:10.356395    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" podStartSLOduration=1.899957407 podStartE2EDuration="2.356381835s" totalImagesPullingTime="456.424428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:27:10.354830729 +0000 UTC m=+2608.702412611" watchObservedRunningTime="2026-07-31 19:27:10.356381835 +0000 UTC m=+2608.703963714"
Jul 31 19:27:10.373811 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:10.373771    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" podStartSLOduration=2.37376229 podStartE2EDuration="2.37376229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:27:10.371841975 +0000 UTC m=+2608.719423861" watchObservedRunningTime="2026-07-31 19:27:10.37376229 +0000 UTC m=+2608.721344170"
Jul 31 19:27:11.344041 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:11.344014    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:16.349730 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:16.349699    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:27:20.340467 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:20.340411    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:27:22.352994 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:22.352966    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:27:42.496435 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:42.496404    2557 scope.go:117] "RemoveContainer" containerID="a650dd00b4a9fedfa6d2a8443571258bc10bd4596611029dcdc7ca83efcd37be"
Jul 31 19:27:50.173803 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.173766    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:27:50.174326 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.174088    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" podUID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" containerName="maas-api" containerID="cri-o://9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36" gracePeriod=30
Jul 31 19:27:50.239422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.239379    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:27:50.239692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.239652    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" podUID="6c4533d9-89ee-44f7-8523-798a594e21a6" containerName="payload-processing" containerID="cri-o://00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c" gracePeriod=30
Jul 31 19:27:50.487388 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.487364    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:27:50.513337 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.513313    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:27:50.543286 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.543251    2557 generic.go:356] "Generic (PLEG): container finished" podID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" containerID="9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36" exitCode=0
Jul 31 19:27:50.543487 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.543289    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"
Jul 31 19:27:50.543487 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.543289    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" event={"ID":"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49","Type":"ContainerDied","Data":"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"}
Jul 31 19:27:50.543487 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.543353    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2" event={"ID":"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49","Type":"ContainerDied","Data":"bf7443a139ccf2077e7448260741900a59508479c4ff7d6eaefe4bd330798bd1"}
Jul 31 19:27:50.543487 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.543387    2557 scope.go:117] "RemoveContainer" containerID="9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"
Jul 31 19:27:50.544714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.544692    2557 generic.go:356] "Generic (PLEG): container finished" podID="6c4533d9-89ee-44f7-8523-798a594e21a6" containerID="00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c" exitCode=0
Jul 31 19:27:50.544813 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.544750    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"
Jul 31 19:27:50.544813 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.544764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" event={"ID":"6c4533d9-89ee-44f7-8523-798a594e21a6","Type":"ContainerDied","Data":"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"}
Jul 31 19:27:50.544813 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.544805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw" event={"ID":"6c4533d9-89ee-44f7-8523-798a594e21a6","Type":"ContainerDied","Data":"6a9d59b45c1f50a095cfecdb2b8c676a69eb6f6c93d6cb8845d5597989b9922e"}
Jul 31 19:27:50.551951 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.551935    2557 scope.go:117] "RemoveContainer" containerID="9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"
Jul 31 19:27:50.552199 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:27:50.552180    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36\": container with ID starting with 9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36 not found: ID does not exist" containerID="9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"
Jul 31 19:27:50.552248 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.552209    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36"} err="failed to get container status \"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36\": rpc error: code = NotFound desc = could not find container \"9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36\": container with ID starting with 9735b5ed4874e72a6922764864c03ad973abaa98dbae6dac37de3fedc3264d36 not found: ID does not exist"
Jul 31 19:27:50.552248 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.552225    2557 scope.go:117] "RemoveContainer" containerID="00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"
Jul 31 19:27:50.561305 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.561285    2557 scope.go:117] "RemoveContainer" containerID="00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"
Jul 31 19:27:50.561667 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:27:50.561644    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c\": container with ID starting with 00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c not found: ID does not exist" containerID="00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"
Jul 31 19:27:50.561735 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.561678    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c"} err="failed to get container status \"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c\": rpc error: code = NotFound desc = could not find container \"00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c\": container with ID starting with 00be8c2fed73447c33c9a182f910d5c406df2176d2cc960f67adc86d434fdd5c not found: ID does not exist"
Jul 31 19:27:50.564925 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.564906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca\") pod \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") "
Jul 31 19:27:50.565014 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.564954    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") pod \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") "
Jul 31 19:27:50.565014 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.564990    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwkx\" (UniqueName: \"kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx\") pod \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\" (UID: \"f7e32cd6-1b03-49a9-b2e1-6d0aef902e49\") "
Jul 31 19:27:50.565285 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.565258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" (UID: "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:27:50.567090 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.567065    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" (UID: "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:27:50.567170 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.567107    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx" (OuterVolumeSpecName: "kube-api-access-xpwkx") pod "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" (UID: "f7e32cd6-1b03-49a9-b2e1-6d0aef902e49"). InnerVolumeSpecName "kube-api-access-xpwkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:27:50.666068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666038    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzm6r\" (UniqueName: \"kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r\") pod \"6c4533d9-89ee-44f7-8523-798a594e21a6\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") "
Jul 31 19:27:50.666272 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume\") pod \"6c4533d9-89ee-44f7-8523-798a594e21a6\" (UID: \"6c4533d9-89ee-44f7-8523-798a594e21a6\") "
Jul 31 19:27:50.666387 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666372    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:27:50.666461 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666394    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:27:50.666461 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666408    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwkx\" (UniqueName: \"kubernetes.io/projected/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49-kube-api-access-xpwkx\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:27:50.666538 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.666433    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c4533d9-89ee-44f7-8523-798a594e21a6" (UID: "6c4533d9-89ee-44f7-8523-798a594e21a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:27:50.668187 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.668166    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r" (OuterVolumeSpecName: "kube-api-access-rzm6r") pod "6c4533d9-89ee-44f7-8523-798a594e21a6" (UID: "6c4533d9-89ee-44f7-8523-798a594e21a6"). InnerVolumeSpecName "kube-api-access-rzm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:27:50.767032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.766950    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzm6r\" (UniqueName: \"kubernetes.io/projected/6c4533d9-89ee-44f7-8523-798a594e21a6-kube-api-access-rzm6r\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:27:50.767032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.766975    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4533d9-89ee-44f7-8523-798a594e21a6-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:27:50.868915 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.868884    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:27:50.876436 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.876410    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968afa-57b7589df9-km5h2"]
Jul 31 19:27:50.888199 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.888179    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:27:50.891731 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:50.891710    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968afa-864f9f4cf4-9rmjw"]
Jul 31 19:27:52.183644 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:52.183609    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4533d9-89ee-44f7-8523-798a594e21a6" path="/var/lib/kubelet/pods/6c4533d9-89ee-44f7-8523-798a594e21a6/volumes"
Jul 31 19:27:52.184018 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:52.183980    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" path="/var/lib/kubelet/pods/f7e32cd6-1b03-49a9-b2e1-6d0aef902e49/volumes"
Jul 31 19:27:57.646949 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.646901    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"]
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647251    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" containerName="maas-api"
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647263    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" containerName="maas-api"
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647277    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4533d9-89ee-44f7-8523-798a594e21a6" containerName="payload-processing"
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647283    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4533d9-89ee-44f7-8523-798a594e21a6" containerName="payload-processing"
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647340    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4533d9-89ee-44f7-8523-798a594e21a6" containerName="payload-processing"
Jul 31 19:27:57.647360 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.647348    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e32cd6-1b03-49a9-b2e1-6d0aef902e49" containerName="maas-api"
Jul 31 19:27:57.651654 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.651638    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:27:57.654566 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.654548    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:27:57.661387 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.661356    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"]
Jul 31 19:27:57.828368 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.828322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz\" (UID: \"a165f0be-9ddc-4ac4-9926-53fe739e5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:27:57.929192 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.929090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz\" (UID: \"a165f0be-9ddc-4ac4-9926-53fe739e5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:27:57.929759 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.929740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz\" (UID: \"a165f0be-9ddc-4ac4-9926-53fe739e5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:27:57.961624 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:57.961600    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:27:58.100975 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:58.100951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"]
Jul 31 19:27:58.102501 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:27:58.102470    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda165f0be_9ddc_4ac4_9926_53fe739e5651.slice/crio-044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e WatchSource:0}: Error finding container 044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e: Status 404 returned error can't find the container with id 044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e
Jul 31 19:27:58.597916 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:58.597852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz" event={"ID":"a165f0be-9ddc-4ac4-9926-53fe739e5651","Type":"ContainerStarted","Data":"3eca3c8ff8cc9e6d1933d8c735d7bd9944e645995c8d3ff4b89831565f347603"}
Jul 31 19:27:58.598047 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:58.597935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz" event={"ID":"a165f0be-9ddc-4ac4-9926-53fe739e5651","Type":"ContainerStarted","Data":"044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e"}
Jul 31 19:27:58.617340 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:58.617286    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz" podStartSLOduration=1.6172713669999998 podStartE2EDuration="1.617271367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:27:58.615971929 +0000 UTC m=+2656.963553810" watchObservedRunningTime="2026-07-31 19:27:58.617271367 +0000 UTC m=+2656.964853277"
Jul 31 19:27:59.605029 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:59.604989    2557 generic.go:356] "Generic (PLEG): container finished" podID="a165f0be-9ddc-4ac4-9926-53fe739e5651" containerID="3eca3c8ff8cc9e6d1933d8c735d7bd9944e645995c8d3ff4b89831565f347603" exitCode=0
Jul 31 19:27:59.605409 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:27:59.605074    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz" event={"ID":"a165f0be-9ddc-4ac4-9926-53fe739e5651","Type":"ContainerDied","Data":"3eca3c8ff8cc9e6d1933d8c735d7bd9944e645995c8d3ff4b89831565f347603"}
Jul 31 19:28:00.742524 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:00.742499    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:28:00.853235 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:00.853198    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca\") pod \"a165f0be-9ddc-4ac4-9926-53fe739e5651\" (UID: \"a165f0be-9ddc-4ac4-9926-53fe739e5651\") "
Jul 31 19:28:00.853618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:00.853593    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a165f0be-9ddc-4ac4-9926-53fe739e5651" (UID: "a165f0be-9ddc-4ac4-9926-53fe739e5651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:28:00.953925 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:00.953840    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a165f0be-9ddc-4ac4-9926-53fe739e5651-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:01.616659 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:01.616622    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz" event={"ID":"a165f0be-9ddc-4ac4-9926-53fe739e5651","Type":"ContainerDied","Data":"044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e"}
Jul 31 19:28:01.616659 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:01.616653    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"
Jul 31 19:28:01.616869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:01.616673    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044b59e07ecae278d34936d581ddd84b054624657756e59f0dc6ac636994fc4e"
Jul 31 19:28:13.361426 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.361392    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:28:13.361937 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.361652    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" podUID="a7364346-6497-46d7-a35a-2321b7455977" containerName="maas-api" containerID="cri-o://2cc8344d477cb1117fc43e2c2c9adcdaf3ed643ad5294843bacbcd94ba3f6d2e" gracePeriod=30
Jul 31 19:28:13.421026 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.420989    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:28:13.421265 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.421217    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" podUID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" containerName="payload-processing" containerID="cri-o://08b84a455fc4609983d79ae17363c0679f317daad455667d735ac1f063a6710b" gracePeriod=30
Jul 31 19:28:13.429583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.429509    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:28:13.429795 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.429731    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" podUID="d7309ce6-16aa-412a-81b4-ec82687bf497" containerName="payload-pre-processing" containerID="cri-o://413e4ae3d2a73dff1fe89db2972c5b659a41b86f553e307ccde490b07e67108e" gracePeriod=30
Jul 31 19:28:13.626740 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.626624    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"]
Jul 31 19:28:13.629826 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.629795    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3dafe-7trjz"]
Jul 31 19:28:13.677046 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.677015    2557 generic.go:356] "Generic (PLEG): container finished" podID="a7364346-6497-46d7-a35a-2321b7455977" containerID="2cc8344d477cb1117fc43e2c2c9adcdaf3ed643ad5294843bacbcd94ba3f6d2e" exitCode=0
Jul 31 19:28:13.677232 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.677094    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" event={"ID":"a7364346-6497-46d7-a35a-2321b7455977","Type":"ContainerDied","Data":"2cc8344d477cb1117fc43e2c2c9adcdaf3ed643ad5294843bacbcd94ba3f6d2e"}
Jul 31 19:28:13.679070 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.679042    2557 generic.go:356] "Generic (PLEG): container finished" podID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" containerID="08b84a455fc4609983d79ae17363c0679f317daad455667d735ac1f063a6710b" exitCode=0
Jul 31 19:28:13.679270 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.679107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" event={"ID":"6ebd8091-bfcc-434b-a44c-fb826e7f01c5","Type":"ContainerDied","Data":"08b84a455fc4609983d79ae17363c0679f317daad455667d735ac1f063a6710b"}
Jul 31 19:28:13.681071 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.681048    2557 generic.go:356] "Generic (PLEG): container finished" podID="d7309ce6-16aa-412a-81b4-ec82687bf497" containerID="413e4ae3d2a73dff1fe89db2972c5b659a41b86f553e307ccde490b07e67108e" exitCode=0
Jul 31 19:28:13.681218 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.681095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" event={"ID":"d7309ce6-16aa-412a-81b4-ec82687bf497","Type":"ContainerDied","Data":"413e4ae3d2a73dff1fe89db2972c5b659a41b86f553e307ccde490b07e67108e"}
Jul 31 19:28:13.705330 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.705306    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:28:13.714359 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.714329    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:28:13.724005 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.723981    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:28:13.759993 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.759686    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume\") pod \"d7309ce6-16aa-412a-81b4-ec82687bf497\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7309ce6-16aa-412a-81b4-ec82687bf497" (UID: "d7309ce6-16aa-412a-81b4-ec82687bf497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760238    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") pod \"a7364346-6497-46d7-a35a-2321b7455977\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760277    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4mfj\" (UniqueName: \"kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj\") pod \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760312    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume\") pod \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\" (UID: \"6ebd8091-bfcc-434b-a44c-fb826e7f01c5\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760364    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grkd5\" (UniqueName: \"kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5\") pod \"d7309ce6-16aa-412a-81b4-ec82687bf497\" (UID: \"d7309ce6-16aa-412a-81b4-ec82687bf497\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760394    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca\") pod \"a7364346-6497-46d7-a35a-2321b7455977\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760459    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdvpc\" (UniqueName: \"kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc\") pod \"a7364346-6497-46d7-a35a-2321b7455977\" (UID: \"a7364346-6497-46d7-a35a-2321b7455977\") "
Jul 31 19:28:13.760745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.760716    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7309ce6-16aa-412a-81b4-ec82687bf497-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.761875 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.761243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ebd8091-bfcc-434b-a44c-fb826e7f01c5" (UID: "6ebd8091-bfcc-434b-a44c-fb826e7f01c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:28:13.761875 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.761364    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7364346-6497-46d7-a35a-2321b7455977" (UID: "a7364346-6497-46d7-a35a-2321b7455977"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:28:13.764002 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.763975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7364346-6497-46d7-a35a-2321b7455977" (UID: "a7364346-6497-46d7-a35a-2321b7455977"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:28:13.764002 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.763975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5" (OuterVolumeSpecName: "kube-api-access-grkd5") pod "d7309ce6-16aa-412a-81b4-ec82687bf497" (UID: "d7309ce6-16aa-412a-81b4-ec82687bf497"). InnerVolumeSpecName "kube-api-access-grkd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:28:13.764185 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.764163    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc" (OuterVolumeSpecName: "kube-api-access-tdvpc") pod "a7364346-6497-46d7-a35a-2321b7455977" (UID: "a7364346-6497-46d7-a35a-2321b7455977"). InnerVolumeSpecName "kube-api-access-tdvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:28:13.764341 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.764323    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj" (OuterVolumeSpecName: "kube-api-access-d4mfj") pod "6ebd8091-bfcc-434b-a44c-fb826e7f01c5" (UID: "6ebd8091-bfcc-434b-a44c-fb826e7f01c5"). InnerVolumeSpecName "kube-api-access-d4mfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:28:13.861944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861907    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvpc\" (UniqueName: \"kubernetes.io/projected/a7364346-6497-46d7-a35a-2321b7455977-kube-api-access-tdvpc\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.861944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861939    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7364346-6497-46d7-a35a-2321b7455977-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.861944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861950    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mfj\" (UniqueName: \"kubernetes.io/projected/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-kube-api-access-d4mfj\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.862184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861958    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebd8091-bfcc-434b-a44c-fb826e7f01c5-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.862184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861969    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkd5\" (UniqueName: \"kubernetes.io/projected/d7309ce6-16aa-412a-81b4-ec82687bf497-kube-api-access-grkd5\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:13.862184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:13.861977    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7364346-6497-46d7-a35a-2321b7455977-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:28:14.183321 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.183278    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a165f0be-9ddc-4ac4-9926-53fe739e5651" path="/var/lib/kubelet/pods/a165f0be-9ddc-4ac4-9926-53fe739e5651/volumes"
Jul 31 19:28:14.692678 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.692648    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"
Jul 31 19:28:14.692678 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.692657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r" event={"ID":"d7309ce6-16aa-412a-81b4-ec82687bf497","Type":"ContainerDied","Data":"a4eb51b02fc61eb4f0b6f358ac5217ced2e53fd008c69a2a7c57e7f45b982eed"}
Jul 31 19:28:14.693199 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.692750    2557 scope.go:117] "RemoveContainer" containerID="413e4ae3d2a73dff1fe89db2972c5b659a41b86f553e307ccde490b07e67108e"
Jul 31 19:28:14.694112 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.694050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf" event={"ID":"a7364346-6497-46d7-a35a-2321b7455977","Type":"ContainerDied","Data":"b58b9eb57d0a93642164e4a460a97194a38c03a5d2445a01549464ea522768fe"}
Jul 31 19:28:14.694112 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.694060    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"
Jul 31 19:28:14.695480 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.695436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g" event={"ID":"6ebd8091-bfcc-434b-a44c-fb826e7f01c5","Type":"ContainerDied","Data":"be60b5d4028664e49ded35bd302897ee9f7caa94c12aacc1218a927354604e39"}
Jul 31 19:28:14.695572 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.695491    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"
Jul 31 19:28:14.703127 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.703100    2557 scope.go:117] "RemoveContainer" containerID="2cc8344d477cb1117fc43e2c2c9adcdaf3ed643ad5294843bacbcd94ba3f6d2e"
Jul 31 19:28:14.711032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.711001    2557 scope.go:117] "RemoveContainer" containerID="08b84a455fc4609983d79ae17363c0679f317daad455667d735ac1f063a6710b"
Jul 31 19:28:14.721276 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.721251    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:28:14.725075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.725051    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3dafe-b48bd45d8-nng5g"]
Jul 31 19:28:14.738366 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.738336    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:28:14.740087 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.740067    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3dafe-77758fd85b-jz55r"]
Jul 31 19:28:14.751485 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.751457    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:28:14.756374 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:14.756354    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3dafe-6fbc86d58d-xzmhf"]
Jul 31 19:28:16.183060 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:16.183027    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" path="/var/lib/kubelet/pods/6ebd8091-bfcc-434b-a44c-fb826e7f01c5/volumes"
Jul 31 19:28:16.183425 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:16.183395    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7364346-6497-46d7-a35a-2321b7455977" path="/var/lib/kubelet/pods/a7364346-6497-46d7-a35a-2321b7455977/volumes"
Jul 31 19:28:16.183717 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:16.183705    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7309ce6-16aa-412a-81b4-ec82687bf497" path="/var/lib/kubelet/pods/d7309ce6-16aa-412a-81b4-ec82687bf497/volumes"
Jul 31 19:28:26.899993 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.899950    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900389    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a165f0be-9ddc-4ac4-9926-53fe739e5651" containerName="revoke-keys"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900406    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a165f0be-9ddc-4ac4-9926-53fe739e5651" containerName="revoke-keys"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900422    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7309ce6-16aa-412a-81b4-ec82687bf497" containerName="payload-pre-processing"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900432    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7309ce6-16aa-412a-81b4-ec82687bf497" containerName="payload-pre-processing"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900471    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" containerName="payload-processing"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900482    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" containerName="payload-processing"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900498    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7364346-6497-46d7-a35a-2321b7455977" containerName="maas-api"
Jul 31 19:28:26.900583 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900508    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7364346-6497-46d7-a35a-2321b7455977" containerName="maas-api"
Jul 31 19:28:26.901001 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900609    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebd8091-bfcc-434b-a44c-fb826e7f01c5" containerName="payload-processing"
Jul 31 19:28:26.901001 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900621    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7364346-6497-46d7-a35a-2321b7455977" containerName="maas-api"
Jul 31 19:28:26.901001 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900629    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7309ce6-16aa-412a-81b4-ec82687bf497" containerName="payload-pre-processing"
Jul 31 19:28:26.901001 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.900639    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a165f0be-9ddc-4ac4-9926-53fe739e5651" containerName="revoke-keys"
Jul 31 19:28:26.903659 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.903644    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:26.906481 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.906426    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 19:28:26.907579 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.907557    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fd0284-serving-cert\""
Jul 31 19:28:26.907579 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.907569    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hqnff\""
Jul 31 19:28:26.907761 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.907610    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 19:28:26.914468 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.914424    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:28:26.972297 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.972265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:26.972297 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.972293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzr8\" (UniqueName: \"kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:26.972578 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:26.972376    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.073680 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.073647    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.073888 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.073712    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.073888 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.073732    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzr8\" (UniqueName: \"kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.073888 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:28:27.073815    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-serving-cert: secret "maas-api-e2e-shared-a-fd0284-serving-cert" not found
Jul 31 19:28:27.074080 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:28:27.073908    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls podName:44607a0f-fafa-4797-8da7-383bd5aead89 nodeName:}" failed. No retries permitted until 2026-07-31 19:28:27.573888652 +0000 UTC m=+2685.921470528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls") pod "maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" (UID: "44607a0f-fafa-4797-8da7-383bd5aead89") : secret "maas-api-e2e-shared-a-fd0284-serving-cert" not found
Jul 31 19:28:27.074409 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.074390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.084356 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.084332    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzr8\" (UniqueName: \"kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.560223 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.560188    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:28:27.563677 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.563662    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.566349 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.566327    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-fd0284-dockercfg-gbqsp\""
Jul 31 19:28:27.566474 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.566426    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-fd0284\""
Jul 31 19:28:27.573403 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.573378    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:28:27.579486 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.579466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.585762 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.582625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.589457 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.589404    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:28:27.593899 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.593863    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.600679 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.600657    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:28:27.680783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.680742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgcx\" (UniqueName: \"kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.680783 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.680785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.681034 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.680877    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.681034 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.680920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqdp\" (UniqueName: \"kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.781858 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.781822    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqdp\" (UniqueName: \"kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.782033 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.781886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgcx\" (UniqueName: \"kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.782033 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.781906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.782033 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.781937    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.782682 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.782662    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.782725 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.782684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.791422 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.791395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgcx\" (UniqueName: \"kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx\") pod \"payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.791607 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.791591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqdp\" (UniqueName: \"kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp\") pod \"payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.815464 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.815363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:27.875644 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.875617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:27.905838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.905786    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:27.962303 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:27.962272    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:28:27.964672 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:28:27.964118    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44607a0f_fafa_4797_8da7_383bd5aead89.slice/crio-beab3f6671e03d667af8554a2cb8f0448e380cd15328ebc86197d625f6d18d68 WatchSource:0}: Error finding container beab3f6671e03d667af8554a2cb8f0448e380cd15328ebc86197d625f6d18d68: Status 404 returned error can't find the container with id beab3f6671e03d667af8554a2cb8f0448e380cd15328ebc86197d625f6d18d68
Jul 31 19:28:28.039715 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.039668    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:28:28.039873 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:28:28.039682    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7f8b07_57c4_4b9d_a9ae_e880c2d1d449.slice/crio-01acdbb1725273783a129c6e56fe3ee45a73f1f7c163121422f8063ab980f24b WatchSource:0}: Error finding container 01acdbb1725273783a129c6e56fe3ee45a73f1f7c163121422f8063ab980f24b: Status 404 returned error can't find the container with id 01acdbb1725273783a129c6e56fe3ee45a73f1f7c163121422f8063ab980f24b
Jul 31 19:28:28.066495 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.066474    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:28:28.072624 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:28:28.072599    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f60b341_8f40_45ec_9493_e9475116aa9a.slice/crio-0129c75a5ffc8655f172328b9f8c05b97c9c1a8f7889a5ae1522d49977977b27 WatchSource:0}: Error finding container 0129c75a5ffc8655f172328b9f8c05b97c9c1a8f7889a5ae1522d49977977b27: Status 404 returned error can't find the container with id 0129c75a5ffc8655f172328b9f8c05b97c9c1a8f7889a5ae1522d49977977b27
Jul 31 19:28:28.766631 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.766591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" event={"ID":"8f60b341-8f40-45ec-9493-e9475116aa9a","Type":"ContainerStarted","Data":"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"}
Jul 31 19:28:28.766843 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.766659    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:28.766843 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.766702    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" event={"ID":"8f60b341-8f40-45ec-9493-e9475116aa9a","Type":"ContainerStarted","Data":"0129c75a5ffc8655f172328b9f8c05b97c9c1a8f7889a5ae1522d49977977b27"}
Jul 31 19:28:28.768278 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.768251    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" event={"ID":"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449","Type":"ContainerStarted","Data":"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"}
Jul 31 19:28:28.768371 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.768297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" event={"ID":"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449","Type":"ContainerStarted","Data":"01acdbb1725273783a129c6e56fe3ee45a73f1f7c163121422f8063ab980f24b"}
Jul 31 19:28:28.768371 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.768322    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:28.769828 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.769809    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" event={"ID":"44607a0f-fafa-4797-8da7-383bd5aead89","Type":"ContainerStarted","Data":"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"}
Jul 31 19:28:28.769942 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.769843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" event={"ID":"44607a0f-fafa-4797-8da7-383bd5aead89","Type":"ContainerStarted","Data":"beab3f6671e03d667af8554a2cb8f0448e380cd15328ebc86197d625f6d18d68"}
Jul 31 19:28:28.770000 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.769951    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:28.785231 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.785194    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" podStartSLOduration=1.7851817639999998 podStartE2EDuration="1.785181764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:28:28.783488109 +0000 UTC m=+2687.131069989" watchObservedRunningTime="2026-07-31 19:28:28.785181764 +0000 UTC m=+2687.132763644"
Jul 31 19:28:28.800660 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.800621    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" podStartSLOduration=1.800609254 podStartE2EDuration="1.800609254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:28:28.799765163 +0000 UTC m=+2687.147347045" watchObservedRunningTime="2026-07-31 19:28:28.800609254 +0000 UTC m=+2687.148191133"
Jul 31 19:28:28.823713 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:28.823654    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" podStartSLOduration=2.440846555 podStartE2EDuration="2.823639221s" totalImagesPullingTime="382.792666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:28:28.821907933 +0000 UTC m=+2687.169489814" watchObservedRunningTime="2026-07-31 19:28:28.823639221 +0000 UTC m=+2687.171221100"
Jul 31 19:28:34.782115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:34.782033    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:28:39.778235 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:39.778201    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:28:39.778729 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:39.778281    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:28:42.327855 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:42.327824    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:28:42.332480 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:28:42.332454    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:29:00.526052 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.526012    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:29:00.529747 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.529729    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.532491 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.532469    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-09a154-serving-cert\""
Jul 31 19:29:00.539494 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.539470    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:29:00.663235 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.663193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.663411 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.663255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.663411 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.663365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xbd\" (UniqueName: \"kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.764306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.764260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.764306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.764310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.764627 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.764353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xbd\" (UniqueName: \"kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.764627 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:29:00.764499    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-serving-cert: secret "maas-api-e2e-shared-b-09a154-serving-cert" not found
Jul 31 19:29:00.764627 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:29:00.764576    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls podName:a3c5ea88-fb12-4b35-bff2-1bf2b8be4848 nodeName:}" failed. No retries permitted until 2026-07-31 19:29:01.264555663 +0000 UTC m=+2719.612137541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls") pod "maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" (UID: "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848") : secret "maas-api-e2e-shared-b-09a154-serving-cert" not found
Jul 31 19:29:00.764999 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.764978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:00.773574 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:00.773556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xbd\" (UniqueName: \"kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:01.213988 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.213947    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:29:01.217831 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.217806    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.220974 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.220952    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-09a154\""
Jul 31 19:29:01.221236 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.221215    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-09a154-dockercfg-zwm4b\""
Jul 31 19:29:01.232746 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.232720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:29:01.269577 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.269533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:01.270160 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:29:01.270125    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-serving-cert: secret "maas-api-e2e-shared-b-09a154-serving-cert" not found
Jul 31 19:29:01.270287 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:29:01.270206    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls podName:a3c5ea88-fb12-4b35-bff2-1bf2b8be4848 nodeName:}" failed. No retries permitted until 2026-07-31 19:29:02.270189855 +0000 UTC m=+2720.617771713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls") pod "maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" (UID: "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848") : secret "maas-api-e2e-shared-b-09a154-serving-cert" not found
Jul 31 19:29:01.371193 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.371157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhs7\" (UniqueName: \"kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.371370 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.371225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.471752 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.471657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhs7\" (UniqueName: \"kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.471752 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.471726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.472471 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.472430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.481190 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.481171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhs7\" (UniqueName: \"kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7\") pod \"payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.529255 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.529201    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.670002 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.669977    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:29:01.671764 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:29:01.671734    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1761937a_1841_4541_9b38_d3c6d30d50b7.slice/crio-54d32c9c7a1a75795311ed28010a573d00bd469d20f6c4bf52b0cc16a2ba11d6 WatchSource:0}: Error finding container 54d32c9c7a1a75795311ed28010a573d00bd469d20f6c4bf52b0cc16a2ba11d6: Status 404 returned error can't find the container with id 54d32c9c7a1a75795311ed28010a573d00bd469d20f6c4bf52b0cc16a2ba11d6
Jul 31 19:29:01.946333 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.946301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" event={"ID":"1761937a-1841-4541-9b38-d3c6d30d50b7","Type":"ContainerStarted","Data":"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"}
Jul 31 19:29:01.946534 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.946349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" event={"ID":"1761937a-1841-4541-9b38-d3c6d30d50b7","Type":"ContainerStarted","Data":"54d32c9c7a1a75795311ed28010a573d00bd469d20f6c4bf52b0cc16a2ba11d6"}
Jul 31 19:29:01.946534 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.946392    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:29:01.968038 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:01.967994    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" podStartSLOduration=0.967980151 podStartE2EDuration="967.980151ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:29:01.967603768 +0000 UTC m=+2720.315185648" watchObservedRunningTime="2026-07-31 19:29:01.967980151 +0000 UTC m=+2720.315562031"
Jul 31 19:29:02.277228 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:02.277191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:02.279785 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:02.279755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:02.344154 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:02.344095    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:02.479910 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:02.479884    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:29:02.481259 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:29:02.481236    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c5ea88_fb12_4b35_bff2_1bf2b8be4848.slice/crio-e7a1597394332e28d4accd41b6e3a81065381f4819b8de4e85cc5512bb46c912 WatchSource:0}: Error finding container e7a1597394332e28d4accd41b6e3a81065381f4819b8de4e85cc5512bb46c912: Status 404 returned error can't find the container with id e7a1597394332e28d4accd41b6e3a81065381f4819b8de4e85cc5512bb46c912
Jul 31 19:29:02.952522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:02.952484    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" event={"ID":"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848","Type":"ContainerStarted","Data":"e7a1597394332e28d4accd41b6e3a81065381f4819b8de4e85cc5512bb46c912"}
Jul 31 19:29:03.958951 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:03.958911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" event={"ID":"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848","Type":"ContainerStarted","Data":"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"}
Jul 31 19:29:03.959396 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:03.958985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:03.977827 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:03.977776    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" podStartSLOduration=3.474656348 podStartE2EDuration="3.977761493s" totalImagesPullingTime="503.105145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:29:03.977183212 +0000 UTC m=+2722.324765092" watchObservedRunningTime="2026-07-31 19:29:03.977761493 +0000 UTC m=+2722.325343372"
Jul 31 19:29:09.969646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:09.969619    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:29:12.954608 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:29:12.954578    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:30:00.136212 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.136177    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"]
Jul 31 19:30:00.140162 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.140140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"
Jul 31 19:30:00.140325 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.140302    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"]
Jul 31 19:30:00.142858 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.142834    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:30:00.143627 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.143610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"
Jul 31 19:30:00.149164 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.149108    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"]
Jul 31 19:30:00.152180 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.152154    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"]
Jul 31 19:30:00.154744 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.154724    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"
Jul 31 19:30:00.163796 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.163762    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"
Jul 31 19:30:00.310054 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.310024    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"]
Jul 31 19:30:00.312464 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:30:00.312402    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444fca39_3cde_4b8a_b68f_5a4110bb7daf.slice/crio-a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0 WatchSource:0}: Error finding container a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0: Status 404 returned error can't find the container with id a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0
Jul 31 19:30:00.331312 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:00.331289    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"]
Jul 31 19:30:00.333100 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:30:00.333075    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2843ff_7cdf_4e07_9d96_74172e23cae8.slice/crio-a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a WatchSource:0}: Error finding container a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a: Status 404 returned error can't find the container with id a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a
Jul 31 19:30:01.249184 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.249144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx" event={"ID":"444fca39-3cde-4b8a-b68f-5a4110bb7daf","Type":"ContainerStarted","Data":"595eeca6571484650ed2fc39a56a28429a816d76e38057ed5e42cdd4617d6728"}
Jul 31 19:30:01.249699 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.249207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx" event={"ID":"444fca39-3cde-4b8a-b68f-5a4110bb7daf","Type":"ContainerStarted","Data":"a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0"}
Jul 31 19:30:01.250740 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.250715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp" event={"ID":"5f2843ff-7cdf-4e07-9d96-74172e23cae8","Type":"ContainerStarted","Data":"397d4850c2506e285ea30f564f2edd481b30a9d4dfd7812e022ee295c356a62c"}
Jul 31 19:30:01.250836 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.250763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp" event={"ID":"5f2843ff-7cdf-4e07-9d96-74172e23cae8","Type":"ContainerStarted","Data":"a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a"}
Jul 31 19:30:01.267154 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.267108    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx" podStartSLOduration=1.2670938729999999 podStartE2EDuration="1.267093873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:30:01.265266317 +0000 UTC m=+2779.612848199" watchObservedRunningTime="2026-07-31 19:30:01.267093873 +0000 UTC m=+2779.614675752"
Jul 31 19:30:01.282073 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:01.282030    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp" podStartSLOduration=1.282015997 podStartE2EDuration="1.282015997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:30:01.281330396 +0000 UTC m=+2779.628912284" watchObservedRunningTime="2026-07-31 19:30:01.282015997 +0000 UTC m=+2779.629597877"
Jul 31 19:30:02.257711 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:02.257675    2557 generic.go:356] "Generic (PLEG): container finished" podID="444fca39-3cde-4b8a-b68f-5a4110bb7daf" containerID="595eeca6571484650ed2fc39a56a28429a816d76e38057ed5e42cdd4617d6728" exitCode=0
Jul 31 19:30:02.258150 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:02.257761    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx" event={"ID":"444fca39-3cde-4b8a-b68f-5a4110bb7daf","Type":"ContainerDied","Data":"595eeca6571484650ed2fc39a56a28429a816d76e38057ed5e42cdd4617d6728"}
Jul 31 19:30:02.259250 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:02.259226    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f2843ff-7cdf-4e07-9d96-74172e23cae8" containerID="397d4850c2506e285ea30f564f2edd481b30a9d4dfd7812e022ee295c356a62c" exitCode=0
Jul 31 19:30:02.259366 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:02.259322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp" event={"ID":"5f2843ff-7cdf-4e07-9d96-74172e23cae8","Type":"ContainerDied","Data":"397d4850c2506e285ea30f564f2edd481b30a9d4dfd7812e022ee295c356a62c"}
Jul 31 19:30:03.390758 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:03.390734    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"
Jul 31 19:30:03.433342 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:03.433313    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"
Jul 31 19:30:04.271701 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.271673    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"
Jul 31 19:30:04.271927 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.271682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx" event={"ID":"444fca39-3cde-4b8a-b68f-5a4110bb7daf","Type":"ContainerDied","Data":"a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0"}
Jul 31 19:30:04.271927 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.271802    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9cdc79ed1cd11a99f82d853ae05ead46ac0b82b6a9e9c1f660b5477457e6cb0"
Jul 31 19:30:04.272954 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.272928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp" event={"ID":"5f2843ff-7cdf-4e07-9d96-74172e23cae8","Type":"ContainerDied","Data":"a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a"}
Jul 31 19:30:04.272954 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.272952    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"
Jul 31 19:30:04.273149 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:30:04.272967    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82a6555285117f158f85eccc57d9c960fafc184eb6f18e07c7ec0855fca724a"
Jul 31 19:32:42.565929 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:32:42.565888    2557 scope.go:117] "RemoveContainer" containerID="173ed1fc38278a11f87a51a8dec702e89bf7fa9de6427eaa6201bfc2ffbe4134"
Jul 31 19:32:42.582208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:32:42.582183    2557 scope.go:117] "RemoveContainer" containerID="d29ee568f9ac1d775a78bb034ac0f1ecf5f0570ffb185e7d7925da36024bcc53"
Jul 31 19:33:42.367099 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:33:42.367068    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:33:42.373785 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:33:42.373759    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:34:42.611285 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:34:42.611193    2557 scope.go:117] "RemoveContainer" containerID="3eca3c8ff8cc9e6d1933d8c735d7bd9944e645995c8d3ff4b89831565f347603"
Jul 31 19:38:31.551735 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.551697    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:38:31.552253 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552238    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444fca39-3cde-4b8a-b68f-5a4110bb7daf" containerName="cleanup"
Jul 31 19:38:31.552306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552258    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="444fca39-3cde-4b8a-b68f-5a4110bb7daf" containerName="cleanup"
Jul 31 19:38:31.552306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552278    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2843ff-7cdf-4e07-9d96-74172e23cae8" containerName="cleanup"
Jul 31 19:38:31.552306 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552287    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2843ff-7cdf-4e07-9d96-74172e23cae8" containerName="cleanup"
Jul 31 19:38:31.552413 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552364    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="444fca39-3cde-4b8a-b68f-5a4110bb7daf" containerName="cleanup"
Jul 31 19:38:31.552413 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.552378    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2843ff-7cdf-4e07-9d96-74172e23cae8" containerName="cleanup"
Jul 31 19:38:31.555616 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.555596    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.558065 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.558045    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0368ff-serving-cert\""
Jul 31 19:38:31.565246 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.565220    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:38:31.692753 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.692713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.692918 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.692775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.692918 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.692837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmnz\" (UniqueName: \"kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.794115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.794075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.794304 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.794146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmnz\" (UniqueName: \"kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.794304 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.794213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.794401 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:38:31.794334    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-serving-cert: secret "maas-api-e2e-ipp-a-0368ff-serving-cert" not found
Jul 31 19:38:31.794473 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:38:31.794420    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls podName:d130de60-1daf-4b90-b252-f985e4521339 nodeName:}" failed. No retries permitted until 2026-07-31 19:38:32.294391682 +0000 UTC m=+3290.641973577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls") pod "maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" (UID: "d130de60-1daf-4b90-b252-f985e4521339") : secret "maas-api-e2e-ipp-a-0368ff-serving-cert" not found
Jul 31 19:38:31.794771 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.794751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:31.813684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:31.813604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmnz\" (UniqueName: \"kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:32.298105 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.298070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:32.300823 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.300799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:32.466372 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.466329    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:32.602616 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.601133    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:38:32.604883 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.604855    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 19:38:32.805486 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:32.805383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" event={"ID":"d130de60-1daf-4b90-b252-f985e4521339","Type":"ContainerStarted","Data":"ac339a5842c6faf567e92464767bd5d232b7b34068b1161db040a475912d1e32"}
Jul 31 19:38:33.811805 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:33.811766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" event={"ID":"d130de60-1daf-4b90-b252-f985e4521339","Type":"ContainerStarted","Data":"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"}
Jul 31 19:38:33.812228 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:33.811908    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:33.831555 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:33.831505    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" podStartSLOduration=2.177581004 podStartE2EDuration="2.831487815s" totalImagesPullingTime="653.906811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:38:33.828750867 +0000 UTC m=+3292.176332749" watchObservedRunningTime="2026-07-31 19:38:33.831487815 +0000 UTC m=+3292.179069694"
Jul 31 19:38:39.822843 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:39.822812    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:38:42.400069 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:42.399972    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:38:42.413759 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:38:42.407214    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:39:05.699105 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.699019    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:39:05.703052 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.703029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.705847 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.705820    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-90f898-serving-cert\""
Jul 31 19:39:05.715133 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.715107    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:39:05.790024 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.789993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpld\" (UniqueName: \"kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.790208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.790047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.790208 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.790094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.890989 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.890942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.891232 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.891030    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.891232 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.891086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpld\" (UniqueName: \"kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.891719 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.891691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.893601 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.893581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:05.900944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:05.900919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpld\" (UniqueName: \"kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld\") pod \"maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:06.016039 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.015947    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:06.161865 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.161814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:39:06.163144 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:39:06.163114    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a23b83a_ac67_4fed_bf0c_ecb22f07bb49.slice/crio-95a20352eecfa8953cfdb540b88fe64f6d3797658e7d3612ccce35d660ba92e1 WatchSource:0}: Error finding container 95a20352eecfa8953cfdb540b88fe64f6d3797658e7d3612ccce35d660ba92e1: Status 404 returned error can't find the container with id 95a20352eecfa8953cfdb540b88fe64f6d3797658e7d3612ccce35d660ba92e1
Jul 31 19:39:06.976989 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.976956    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" event={"ID":"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49","Type":"ContainerStarted","Data":"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"}
Jul 31 19:39:06.977412 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.977035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" event={"ID":"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49","Type":"ContainerStarted","Data":"95a20352eecfa8953cfdb540b88fe64f6d3797658e7d3612ccce35d660ba92e1"}
Jul 31 19:39:06.977412 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.977063    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:39:06.996219 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:06.996175    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" podStartSLOduration=1.590290404 podStartE2EDuration="1.996160361s" totalImagesPullingTime="405.869957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:39:06.995263499 +0000 UTC m=+3325.342845390" watchObservedRunningTime="2026-07-31 19:39:06.996160361 +0000 UTC m=+3325.343742242"
Jul 31 19:39:12.987939 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:39:12.987912    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:40:21.082218 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.082173    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:21.085716 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.085696    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.090716 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.090696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-85e026-serving-cert\""
Jul 31 19:40:21.092519 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.092496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.092620 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.092598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxbd\" (UniqueName: \"kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.092685 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.092629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.108971 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.108879    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:21.193124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.193087    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxbd\" (UniqueName: \"kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.193124 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.193122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.193378 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.193153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.193378 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:40:21.193242    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-serving-cert: secret "maas-api-e2e-ipp-cleanup-85e026-serving-cert" not found
Jul 31 19:40:21.193378 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:40:21.193316    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls podName:d35b12ad-fb3c-4ea6-8852-f26dcdd522f1 nodeName:}" failed. No retries permitted until 2026-07-31 19:40:21.693300825 +0000 UTC m=+3400.040882686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" (UID: "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1") : secret "maas-api-e2e-ipp-cleanup-85e026-serving-cert" not found
Jul 31 19:40:21.193759 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.193740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.202388 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.202360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxbd\" (UniqueName: \"kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.697668 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.697635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.700131 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.700103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:21.757825 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.757792    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:21.761270 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.761255    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.764075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.764038    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-85e026\""
Jul 31 19:40:21.764194 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.764175    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-85e026-dockercfg-clkcm\""
Jul 31 19:40:21.772288 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.772267    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:21.798941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.798917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4m8\" (UniqueName: \"kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.799095 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.798976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.899584 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.899543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4m8\" (UniqueName: \"kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.899769 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.899618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.900351 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.900333    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.909615 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.909588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4m8\" (UniqueName: \"kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8\") pod \"payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:21.997499 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:21.997419    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:22.072687 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.072650    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:22.147944 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.147919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:22.150130 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:40:22.150101    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35b12ad_fb3c_4ea6_8852_f26dcdd522f1.slice/crio-e4e76678138c0dd7e44e036b17c423ef0cb9f3b7a1b152f1aa37d9d0600663e3 WatchSource:0}: Error finding container e4e76678138c0dd7e44e036b17c423ef0cb9f3b7a1b152f1aa37d9d0600663e3: Status 404 returned error can't find the container with id e4e76678138c0dd7e44e036b17c423ef0cb9f3b7a1b152f1aa37d9d0600663e3
Jul 31 19:40:22.218179 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.218145    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:22.219347 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:40:22.219322    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f684003_b613_4d66_b5b2_71d45356e976.slice/crio-0780e5eeb68e8d0f7f2032c6503f649d8a8cbbcd09bd5cda779d68b8dcfbf22f WatchSource:0}: Error finding container 0780e5eeb68e8d0f7f2032c6503f649d8a8cbbcd09bd5cda779d68b8dcfbf22f: Status 404 returned error can't find the container with id 0780e5eeb68e8d0f7f2032c6503f649d8a8cbbcd09bd5cda779d68b8dcfbf22f
Jul 31 19:40:22.350800 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.350760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" event={"ID":"3f684003-b613-4d66-b5b2-71d45356e976","Type":"ContainerStarted","Data":"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"}
Jul 31 19:40:22.350981 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.350824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" event={"ID":"3f684003-b613-4d66-b5b2-71d45356e976","Type":"ContainerStarted","Data":"0780e5eeb68e8d0f7f2032c6503f649d8a8cbbcd09bd5cda779d68b8dcfbf22f"}
Jul 31 19:40:22.350981 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.350856    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:22.352127 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.352096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" event={"ID":"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1","Type":"ContainerStarted","Data":"e4e76678138c0dd7e44e036b17c423ef0cb9f3b7a1b152f1aa37d9d0600663e3"}
Jul 31 19:40:22.368119 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:22.368066    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" podStartSLOduration=1.368046128 podStartE2EDuration="1.368046128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:40:22.366642103 +0000 UTC m=+3400.714223985" watchObservedRunningTime="2026-07-31 19:40:22.368046128 +0000 UTC m=+3400.715628009"
Jul 31 19:40:23.359539 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:23.359500    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" event={"ID":"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1","Type":"ContainerStarted","Data":"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"}
Jul 31 19:40:23.359997 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:23.359636    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:23.379896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:23.379849    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" podStartSLOduration=1.951589163 podStartE2EDuration="2.379833973s" totalImagesPullingTime="428.24481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:40:23.377949866 +0000 UTC m=+3401.725531747" watchObservedRunningTime="2026-07-31 19:40:23.379833973 +0000 UTC m=+3401.727415843"
Jul 31 19:40:29.370284 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:29.370247    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:33.361626 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:33.361593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:45.620860 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.620822    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"]
Jul 31 19:40:45.624347 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.624331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:45.627889 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.627865    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:40:45.637874 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.637844    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"]
Jul 31 19:40:45.703905 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.703869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7\" (UID: \"8c405b55-43fd-455c-b141-3c390094c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:45.805013 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.804975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7\" (UID: \"8c405b55-43fd-455c-b141-3c390094c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:45.805636 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.805618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7\" (UID: \"8c405b55-43fd-455c-b141-3c390094c9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:45.933995 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:45.933907    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:46.071919 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:46.071894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"]
Jul 31 19:40:46.073838 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:40:46.073808    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c405b55_43fd_455c_b141_3c390094c9de.slice/crio-d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a WatchSource:0}: Error finding container d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a: Status 404 returned error can't find the container with id d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a
Jul 31 19:40:46.480381 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:46.480350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7" event={"ID":"8c405b55-43fd-455c-b141-3c390094c9de","Type":"ContainerStarted","Data":"084549cb0a993b62f4f042fed01f46f2ffcec5fcee3c7ce366013a7e6b6f468d"}
Jul 31 19:40:46.480575 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:46.480398    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7" event={"ID":"8c405b55-43fd-455c-b141-3c390094c9de","Type":"ContainerStarted","Data":"d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a"}
Jul 31 19:40:46.498366 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:46.498322    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7" podStartSLOduration=1.498308529 podStartE2EDuration="1.498308529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:40:46.496626178 +0000 UTC m=+3424.844208060" watchObservedRunningTime="2026-07-31 19:40:46.498308529 +0000 UTC m=+3424.845890413"
Jul 31 19:40:47.486721 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:47.486687    2557 generic.go:356] "Generic (PLEG): container finished" podID="8c405b55-43fd-455c-b141-3c390094c9de" containerID="084549cb0a993b62f4f042fed01f46f2ffcec5fcee3c7ce366013a7e6b6f468d" exitCode=0
Jul 31 19:40:47.487122 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:47.486776    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7" event={"ID":"8c405b55-43fd-455c-b141-3c390094c9de","Type":"ContainerDied","Data":"084549cb0a993b62f4f042fed01f46f2ffcec5fcee3c7ce366013a7e6b6f468d"}
Jul 31 19:40:48.629314 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:48.629291    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:48.734427 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:48.734389    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca\") pod \"8c405b55-43fd-455c-b141-3c390094c9de\" (UID: \"8c405b55-43fd-455c-b141-3c390094c9de\") "
Jul 31 19:40:48.734790 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:48.734767    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c405b55-43fd-455c-b141-3c390094c9de" (UID: "8c405b55-43fd-455c-b141-3c390094c9de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:40:48.835851 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:48.835758    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c405b55-43fd-455c-b141-3c390094c9de-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:49.498924 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:49.498885    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7" event={"ID":"8c405b55-43fd-455c-b141-3c390094c9de","Type":"ContainerDied","Data":"d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a"}
Jul 31 19:40:49.499111 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:49.498945    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28d0f2c8441086bf5e58b8bbcdd92ca61fc364a75ef6166302e80678470aa0a"
Jul 31 19:40:49.499111 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:49.498905    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"
Jul 31 19:40:51.778848 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.778806    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:51.779307 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.779083    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" podUID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" containerName="maas-api" containerID="cri-o://5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127" gracePeriod=30
Jul 31 19:40:51.840790 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.840752    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:51.841045 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.841020    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" podUID="3f684003-b613-4d66-b5b2-71d45356e976" containerName="payload-processing" containerID="cri-o://b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6" gracePeriod=30
Jul 31 19:40:51.991127 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.991091    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"]
Jul 31 19:40:51.993873 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:51.993826    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85e026-wnnf7"]
Jul 31 19:40:52.103386 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.103358    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:52.130132 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.130108    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:52.167085 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.167059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca\") pod \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") "
Jul 31 19:40:52.167250 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.167119    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfxbd\" (UniqueName: \"kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd\") pod \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") "
Jul 31 19:40:52.167250 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.167142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") pod \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\" (UID: \"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1\") "
Jul 31 19:40:52.167507 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.167483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" (UID: "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:40:52.169398 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.169366    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" (UID: "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:40:52.169526 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.169419    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd" (OuterVolumeSpecName: "kube-api-access-wfxbd") pod "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" (UID: "d35b12ad-fb3c-4ea6-8852-f26dcdd522f1"). InnerVolumeSpecName "kube-api-access-wfxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:40:52.183507 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.183483    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c405b55-43fd-455c-b141-3c390094c9de" path="/var/lib/kubelet/pods/8c405b55-43fd-455c-b141-3c390094c9de/volumes"
Jul 31 19:40:52.268045 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume\") pod \"3f684003-b613-4d66-b5b2-71d45356e976\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") "
Jul 31 19:40:52.268236 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268095    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd4m8\" (UniqueName: \"kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8\") pod \"3f684003-b613-4d66-b5b2-71d45356e976\" (UID: \"3f684003-b613-4d66-b5b2-71d45356e976\") "
Jul 31 19:40:52.268354 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268336    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfxbd\" (UniqueName: \"kubernetes.io/projected/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-kube-api-access-wfxbd\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:52.268419 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268359    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:52.268419 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268375    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:52.268548 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.268485    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f684003-b613-4d66-b5b2-71d45356e976" (UID: "3f684003-b613-4d66-b5b2-71d45356e976"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:40:52.270309 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.270291    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8" (OuterVolumeSpecName: "kube-api-access-gd4m8") pod "3f684003-b613-4d66-b5b2-71d45356e976" (UID: "3f684003-b613-4d66-b5b2-71d45356e976"). InnerVolumeSpecName "kube-api-access-gd4m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:40:52.369692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.369641    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4m8\" (UniqueName: \"kubernetes.io/projected/3f684003-b613-4d66-b5b2-71d45356e976-kube-api-access-gd4m8\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:52.369692 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.369687    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f684003-b613-4d66-b5b2-71d45356e976-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:40:52.516218 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.516176    2557 generic.go:356] "Generic (PLEG): container finished" podID="3f684003-b613-4d66-b5b2-71d45356e976" containerID="b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6" exitCode=0
Jul 31 19:40:52.516418 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.516226    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"
Jul 31 19:40:52.516418 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.516265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" event={"ID":"3f684003-b613-4d66-b5b2-71d45356e976","Type":"ContainerDied","Data":"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"}
Jul 31 19:40:52.516418 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.516321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9" event={"ID":"3f684003-b613-4d66-b5b2-71d45356e976","Type":"ContainerDied","Data":"0780e5eeb68e8d0f7f2032c6503f649d8a8cbbcd09bd5cda779d68b8dcfbf22f"}
Jul 31 19:40:52.516418 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.516353    2557 scope.go:117] "RemoveContainer" containerID="b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"
Jul 31 19:40:52.517722 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.517697    2557 generic.go:356] "Generic (PLEG): container finished" podID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" containerID="5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127" exitCode=0
Jul 31 19:40:52.517814 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.517743    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"
Jul 31 19:40:52.517814 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.517771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" event={"ID":"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1","Type":"ContainerDied","Data":"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"}
Jul 31 19:40:52.518097 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.517817    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d" event={"ID":"d35b12ad-fb3c-4ea6-8852-f26dcdd522f1","Type":"ContainerDied","Data":"e4e76678138c0dd7e44e036b17c423ef0cb9f3b7a1b152f1aa37d9d0600663e3"}
Jul 31 19:40:52.525885 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.525867    2557 scope.go:117] "RemoveContainer" containerID="b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"
Jul 31 19:40:52.526156 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:40:52.526139    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6\": container with ID starting with b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6 not found: ID does not exist" containerID="b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"
Jul 31 19:40:52.526210 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.526167    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6"} err="failed to get container status \"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6\": rpc error: code = NotFound desc = could not find container \"b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6\": container with ID starting with b10535c4f0dcf12e045b66d8fc7c623fcf371ed53a670b110a56b75785fed2b6 not found: ID does not exist"
Jul 31 19:40:52.526210 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.526185    2557 scope.go:117] "RemoveContainer" containerID="5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"
Jul 31 19:40:52.534700 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.534679    2557 scope.go:117] "RemoveContainer" containerID="5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"
Jul 31 19:40:52.535010 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:40:52.534984    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127\": container with ID starting with 5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127 not found: ID does not exist" containerID="5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"
Jul 31 19:40:52.535075 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.535017    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127"} err="failed to get container status \"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127\": rpc error: code = NotFound desc = could not find container \"5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127\": container with ID starting with 5a1f8e6273ae840de29f873d245e9ef5b97cf99952975dd6f647c4932dc60127 not found: ID does not exist"
Jul 31 19:40:52.535903 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.535882    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:52.539745 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.539722    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85e026-844d9f788-qn27d"]
Jul 31 19:40:52.550519 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.550496    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:52.556361 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:52.556337    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85e026-fdbf8fb4d-b6kh9"]
Jul 31 19:40:54.186747 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:54.184351    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f684003-b613-4d66-b5b2-71d45356e976" path="/var/lib/kubelet/pods/3f684003-b613-4d66-b5b2-71d45356e976/volumes"
Jul 31 19:40:54.186747 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:40:54.184982    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" path="/var/lib/kubelet/pods/d35b12ad-fb3c-4ea6-8852-f26dcdd522f1/volumes"
Jul 31 19:41:02.059070 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059033    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"]
Jul 31 19:41:02.059566 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059540    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" containerName="maas-api"
Jul 31 19:41:02.059566 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059561    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" containerName="maas-api"
Jul 31 19:41:02.059703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059579    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f684003-b613-4d66-b5b2-71d45356e976" containerName="payload-processing"
Jul 31 19:41:02.059703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059587    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f684003-b613-4d66-b5b2-71d45356e976" containerName="payload-processing"
Jul 31 19:41:02.059703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059605    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c405b55-43fd-455c-b141-3c390094c9de" containerName="revoke-keys"
Jul 31 19:41:02.059703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059612    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c405b55-43fd-455c-b141-3c390094c9de" containerName="revoke-keys"
Jul 31 19:41:02.059905 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059715    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c405b55-43fd-455c-b141-3c390094c9de" containerName="revoke-keys"
Jul 31 19:41:02.059905 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059725    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d35b12ad-fb3c-4ea6-8852-f26dcdd522f1" containerName="maas-api"
Jul 31 19:41:02.059905 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.059739    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f684003-b613-4d66-b5b2-71d45356e976" containerName="payload-processing"
Jul 31 19:41:02.066168 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.064880    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:02.068097 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.068065    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:41:02.070295 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.070269    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"]
Jul 31 19:41:02.155886 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.155845    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4\" (UID: \"b886ae98-0037-46e5-9e44-ed49dbad185b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:02.256685 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.256636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4\" (UID: \"b886ae98-0037-46e5-9e44-ed49dbad185b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:02.257255 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.257232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4\" (UID: \"b886ae98-0037-46e5-9e44-ed49dbad185b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:02.379032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.378993    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:02.509871 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.509840    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"]
Jul 31 19:41:02.511734 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:41:02.511699    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb886ae98_0037_46e5_9e44_ed49dbad185b.slice/crio-213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d WatchSource:0}: Error finding container 213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d: Status 404 returned error can't find the container with id 213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d
Jul 31 19:41:02.574137 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:02.574102    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4" event={"ID":"b886ae98-0037-46e5-9e44-ed49dbad185b","Type":"ContainerStarted","Data":"213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d"}
Jul 31 19:41:03.580637 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:03.580600    2557 generic.go:356] "Generic (PLEG): container finished" podID="b886ae98-0037-46e5-9e44-ed49dbad185b" containerID="3fca12c2d77666c7bd71603b4f98f8db4822f4cd673a75ede7adcd019f70c661" exitCode=0
Jul 31 19:41:03.581033 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:03.580690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4" event={"ID":"b886ae98-0037-46e5-9e44-ed49dbad185b","Type":"ContainerDied","Data":"3fca12c2d77666c7bd71603b4f98f8db4822f4cd673a75ede7adcd019f70c661"}
Jul 31 19:41:04.712877 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:04.712851    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:04.878802 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:04.878770    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca\") pod \"b886ae98-0037-46e5-9e44-ed49dbad185b\" (UID: \"b886ae98-0037-46e5-9e44-ed49dbad185b\") "
Jul 31 19:41:04.879099 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:04.879076    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b886ae98-0037-46e5-9e44-ed49dbad185b" (UID: "b886ae98-0037-46e5-9e44-ed49dbad185b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:41:04.979490 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:04.979464    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b886ae98-0037-46e5-9e44-ed49dbad185b-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:41:05.591094 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:05.591064    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"
Jul 31 19:41:05.591094 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:05.591088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4" event={"ID":"b886ae98-0037-46e5-9e44-ed49dbad185b","Type":"ContainerDied","Data":"213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d"}
Jul 31 19:41:05.591304 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:05.591133    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="213641d16a8b27efcd9761ff806f9891205ac0485178f3dd12ce598e1ac5178d"
Jul 31 19:41:11.791714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:11.791673    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:41:11.792143 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:11.791890    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" podUID="d130de60-1daf-4b90-b252-f985e4521339" containerName="maas-api" containerID="cri-o://f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc" gracePeriod=30
Jul 31 19:41:11.999183 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:11.999149    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"]
Jul 31 19:41:12.003541 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.003512    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0368ff-csrq4"]
Jul 31 19:41:12.185052 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.185017    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b886ae98-0037-46e5-9e44-ed49dbad185b" path="/var/lib/kubelet/pods/b886ae98-0037-46e5-9e44-ed49dbad185b/volumes"
Jul 31 19:41:12.241144 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.241122    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:41:12.345487 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.345381    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") pod \"d130de60-1daf-4b90-b252-f985e4521339\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") "
Jul 31 19:41:12.345654 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.345497    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cmnz\" (UniqueName: \"kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz\") pod \"d130de60-1daf-4b90-b252-f985e4521339\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") "
Jul 31 19:41:12.345654 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.345544    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca\") pod \"d130de60-1daf-4b90-b252-f985e4521339\" (UID: \"d130de60-1daf-4b90-b252-f985e4521339\") "
Jul 31 19:41:12.346038 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.345942    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d130de60-1daf-4b90-b252-f985e4521339" (UID: "d130de60-1daf-4b90-b252-f985e4521339"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:41:12.347898 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.347857    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d130de60-1daf-4b90-b252-f985e4521339" (UID: "d130de60-1daf-4b90-b252-f985e4521339"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:41:12.348400 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.348372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz" (OuterVolumeSpecName: "kube-api-access-4cmnz") pod "d130de60-1daf-4b90-b252-f985e4521339" (UID: "d130de60-1daf-4b90-b252-f985e4521339"). InnerVolumeSpecName "kube-api-access-4cmnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:41:12.446688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.446650    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d130de60-1daf-4b90-b252-f985e4521339-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:41:12.446688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.446683    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmnz\" (UniqueName: \"kubernetes.io/projected/d130de60-1daf-4b90-b252-f985e4521339-kube-api-access-4cmnz\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:41:12.446688 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.446693    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d130de60-1daf-4b90-b252-f985e4521339-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:41:12.630949 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.630909    2557 generic.go:356] "Generic (PLEG): container finished" podID="d130de60-1daf-4b90-b252-f985e4521339" containerID="f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc" exitCode=0
Jul 31 19:41:12.630949 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.630953    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"
Jul 31 19:41:12.631191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.630998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" event={"ID":"d130de60-1daf-4b90-b252-f985e4521339","Type":"ContainerDied","Data":"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"}
Jul 31 19:41:12.631191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.631059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd" event={"ID":"d130de60-1daf-4b90-b252-f985e4521339","Type":"ContainerDied","Data":"ac339a5842c6faf567e92464767bd5d232b7b34068b1161db040a475912d1e32"}
Jul 31 19:41:12.631191 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.631082    2557 scope.go:117] "RemoveContainer" containerID="f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"
Jul 31 19:41:12.640914 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.640895    2557 scope.go:117] "RemoveContainer" containerID="f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"
Jul 31 19:41:12.641186 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:41:12.641167    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc\": container with ID starting with f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc not found: ID does not exist" containerID="f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"
Jul 31 19:41:12.641262 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.641192    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc"} err="failed to get container status \"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc\": rpc error: code = NotFound desc = could not find container \"f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc\": container with ID starting with f3df68a7f3a3de1490bf9046e6f92436f9c0320e70c03800fba78f3d68ff04cc not found: ID does not exist"
Jul 31 19:41:12.653784 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.653760    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:41:12.657998 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:12.657976    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0368ff-754fbfcdc6-dzgvd"]
Jul 31 19:41:14.183766 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:41:14.183734    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d130de60-1daf-4b90-b252-f985e4521339" path="/var/lib/kubelet/pods/d130de60-1daf-4b90-b252-f985e4521339/volumes"
Jul 31 19:42:07.087388 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087308    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"]
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087647    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d130de60-1daf-4b90-b252-f985e4521339" containerName="maas-api"
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087661    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d130de60-1daf-4b90-b252-f985e4521339" containerName="maas-api"
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087691    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b886ae98-0037-46e5-9e44-ed49dbad185b" containerName="revoke-keys"
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087697    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b886ae98-0037-46e5-9e44-ed49dbad185b" containerName="revoke-keys"
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087779    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d130de60-1daf-4b90-b252-f985e4521339" containerName="maas-api"
Jul 31 19:42:07.087819 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.087791    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b886ae98-0037-46e5-9e44-ed49dbad185b" containerName="revoke-keys"
Jul 31 19:42:07.090780 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.090766    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:07.093906 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.093887    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:42:07.099244 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.099215    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"]
Jul 31 19:42:07.210229 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.210191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd\" (UID: \"a25672e9-d634-4653-a424-db451f303658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:07.310932 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.310888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd\" (UID: \"a25672e9-d634-4653-a424-db451f303658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:07.311557 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.311539    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd\" (UID: \"a25672e9-d634-4653-a424-db451f303658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:07.402095 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.402061    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:07.540814 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.540790    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"]
Jul 31 19:42:07.542704 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:42:07.542676    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25672e9_d634_4653_a424_db451f303658.slice/crio-343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8 WatchSource:0}: Error finding container 343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8: Status 404 returned error can't find the container with id 343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8
Jul 31 19:42:07.915392 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.915353    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd" event={"ID":"a25672e9-d634-4653-a424-db451f303658","Type":"ContainerStarted","Data":"19f46c17571f6053e7888058f994f78625664278cdb9ea6b1c327302130d7cf3"}
Jul 31 19:42:07.915606 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.915413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd" event={"ID":"a25672e9-d634-4653-a424-db451f303658","Type":"ContainerStarted","Data":"343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8"}
Jul 31 19:42:07.933888 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:07.933829    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd" podStartSLOduration=0.933814252 podStartE2EDuration="933.814252ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:42:07.931521688 +0000 UTC m=+3506.279103570" watchObservedRunningTime="2026-07-31 19:42:07.933814252 +0000 UTC m=+3506.281396164"
Jul 31 19:42:08.927108 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:08.927076    2557 generic.go:356] "Generic (PLEG): container finished" podID="a25672e9-d634-4653-a424-db451f303658" containerID="19f46c17571f6053e7888058f994f78625664278cdb9ea6b1c327302130d7cf3" exitCode=0
Jul 31 19:42:08.927529 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:08.927167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd" event={"ID":"a25672e9-d634-4653-a424-db451f303658","Type":"ContainerDied","Data":"19f46c17571f6053e7888058f994f78625664278cdb9ea6b1c327302130d7cf3"}
Jul 31 19:42:10.060501 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.060477    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:10.237070 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.236980    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca\") pod \"a25672e9-d634-4653-a424-db451f303658\" (UID: \"a25672e9-d634-4653-a424-db451f303658\") "
Jul 31 19:42:10.237369 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.237344    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a25672e9-d634-4653-a424-db451f303658" (UID: "a25672e9-d634-4653-a424-db451f303658"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:10.337584 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.337545    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25672e9-d634-4653-a424-db451f303658-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:10.938941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.938913    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"
Jul 31 19:42:10.939136 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.938912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd" event={"ID":"a25672e9-d634-4653-a424-db451f303658","Type":"ContainerDied","Data":"343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8"}
Jul 31 19:42:10.939136 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:10.939056    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="343b9a34460b87581488dd153e0aa97da68e25e9d6de0f7d81dd383fbf2481c8"
Jul 31 19:42:17.529958 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.529915    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:42:17.530566 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.530190    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" podUID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" containerName="maas-api" containerID="cri-o://d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe" gracePeriod=30
Jul 31 19:42:17.779761 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.779737    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:42:17.799907 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.799830    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls\") pod \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") "
Jul 31 19:42:17.799907 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.799902    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca\") pod \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") "
Jul 31 19:42:17.800189 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.800165    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cpld\" (UniqueName: \"kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld\") pod \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\" (UID: \"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49\") "
Jul 31 19:42:17.800375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.800345    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" (UID: "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:17.800667 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.800580    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:17.802960 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.802912    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld" (OuterVolumeSpecName: "kube-api-access-6cpld") pod "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" (UID: "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49"). InnerVolumeSpecName "kube-api-access-6cpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:17.803171 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.802991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" (UID: "4a23b83a-ac67-4fed-bf0c-ecb22f07bb49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:42:17.804737 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.804714    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"]
Jul 31 19:42:17.806800 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.806779    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90f898-b6vfd"]
Jul 31 19:42:17.901133 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.901078    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:17.901133 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.901115    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpld\" (UniqueName: \"kubernetes.io/projected/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49-kube-api-access-6cpld\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:17.982242 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.982210    2557 generic.go:356] "Generic (PLEG): container finished" podID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" containerID="d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe" exitCode=0
Jul 31 19:42:17.982405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.982257    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"
Jul 31 19:42:17.982405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.982293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" event={"ID":"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49","Type":"ContainerDied","Data":"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"}
Jul 31 19:42:17.982405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.982353    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr" event={"ID":"4a23b83a-ac67-4fed-bf0c-ecb22f07bb49","Type":"ContainerDied","Data":"95a20352eecfa8953cfdb540b88fe64f6d3797658e7d3612ccce35d660ba92e1"}
Jul 31 19:42:17.982405 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.982376    2557 scope.go:117] "RemoveContainer" containerID="d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"
Jul 31 19:42:17.992003 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.991984    2557 scope.go:117] "RemoveContainer" containerID="d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"
Jul 31 19:42:17.992262 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:17.992244    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe\": container with ID starting with d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe not found: ID does not exist" containerID="d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"
Jul 31 19:42:17.992330 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:17.992272    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe"} err="failed to get container status \"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe\": rpc error: code = NotFound desc = could not find container \"d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe\": container with ID starting with d1c1dcc4bd7e222fa3615dbbfc5d95c96ec616781c4685ae3cee2c3ed8861ffe not found: ID does not exist"
Jul 31 19:42:18.005697 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:18.005673    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:42:18.009640 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:18.009613    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90f898-75c7796f4b-d2tcr"]
Jul 31 19:42:18.183366 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:18.183337    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" path="/var/lib/kubelet/pods/4a23b83a-ac67-4fed-bf0c-ecb22f07bb49/volumes"
Jul 31 19:42:18.183684 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:18.183671    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25672e9-d634-4653-a424-db451f303658" path="/var/lib/kubelet/pods/a25672e9-d634-4653-a424-db451f303658/volumes"
Jul 31 19:42:35.398399 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.398360    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:42:35.398965 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.398652    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" podUID="44607a0f-fafa-4797-8da7-383bd5aead89" containerName="maas-api" containerID="cri-o://819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea" gracePeriod=30
Jul 31 19:42:35.430623 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.430575    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"]
Jul 31 19:42:35.432807 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.432786    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd0284-29758770-4lzwp"]
Jul 31 19:42:35.455472 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.455416    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:42:35.455703 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.455655    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" podUID="8f60b341-8f40-45ec-9493-e9475116aa9a" containerName="payload-processing" containerID="cri-o://c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542" gracePeriod=30
Jul 31 19:42:35.465958 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.465794    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:42:35.466044 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.466006    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" podUID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" containerName="payload-pre-processing" containerID="cri-o://547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9" gracePeriod=30
Jul 31 19:42:35.712053 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.712021    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:42:35.735187 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.735154    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zzr8\" (UniqueName: \"kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8\") pod \"44607a0f-fafa-4797-8da7-383bd5aead89\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") "
Jul 31 19:42:35.736049 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.735243    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca\") pod \"44607a0f-fafa-4797-8da7-383bd5aead89\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") "
Jul 31 19:42:35.736049 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.735269    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") pod \"44607a0f-fafa-4797-8da7-383bd5aead89\" (UID: \"44607a0f-fafa-4797-8da7-383bd5aead89\") "
Jul 31 19:42:35.736049 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.736015    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44607a0f-fafa-4797-8da7-383bd5aead89" (UID: "44607a0f-fafa-4797-8da7-383bd5aead89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:35.736280 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.736253    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44607a0f-fafa-4797-8da7-383bd5aead89-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.738990 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.738950    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8" (OuterVolumeSpecName: "kube-api-access-7zzr8") pod "44607a0f-fafa-4797-8da7-383bd5aead89" (UID: "44607a0f-fafa-4797-8da7-383bd5aead89"). InnerVolumeSpecName "kube-api-access-7zzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:35.740963 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.740936    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44607a0f-fafa-4797-8da7-383bd5aead89" (UID: "44607a0f-fafa-4797-8da7-383bd5aead89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:42:35.757483 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.757462    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:42:35.768201 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.768182    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:42:35.836714 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836686    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume\") pod \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") "
Jul 31 19:42:35.836896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836730    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znqdp\" (UniqueName: \"kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp\") pod \"8f60b341-8f40-45ec-9493-e9475116aa9a\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") "
Jul 31 19:42:35.836896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836773    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dgcx\" (UniqueName: \"kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx\") pod \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\" (UID: \"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449\") "
Jul 31 19:42:35.836896 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume\") pod \"8f60b341-8f40-45ec-9493-e9475116aa9a\" (UID: \"8f60b341-8f40-45ec-9493-e9475116aa9a\") "
Jul 31 19:42:35.837076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836968    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzr8\" (UniqueName: \"kubernetes.io/projected/44607a0f-fafa-4797-8da7-383bd5aead89-kube-api-access-7zzr8\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.837076 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.836985    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44607a0f-fafa-4797-8da7-383bd5aead89-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.837175 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.837142    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" (UID: "6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:35.837233 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.837204    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f60b341-8f40-45ec-9493-e9475116aa9a" (UID: "8f60b341-8f40-45ec-9493-e9475116aa9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:35.839033 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.838996    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx" (OuterVolumeSpecName: "kube-api-access-9dgcx") pod "6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" (UID: "6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449"). InnerVolumeSpecName "kube-api-access-9dgcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:35.839089 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.839040    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp" (OuterVolumeSpecName: "kube-api-access-znqdp") pod "8f60b341-8f40-45ec-9493-e9475116aa9a" (UID: "8f60b341-8f40-45ec-9493-e9475116aa9a"). InnerVolumeSpecName "kube-api-access-znqdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:35.938375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.938293    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f60b341-8f40-45ec-9493-e9475116aa9a-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.938375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.938322    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.938375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.938331    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znqdp\" (UniqueName: \"kubernetes.io/projected/8f60b341-8f40-45ec-9493-e9475116aa9a-kube-api-access-znqdp\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:35.938375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:35.938339    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgcx\" (UniqueName: \"kubernetes.io/projected/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449-kube-api-access-9dgcx\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:36.079698 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.079666    2557 generic.go:356] "Generic (PLEG): container finished" podID="8f60b341-8f40-45ec-9493-e9475116aa9a" containerID="c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542" exitCode=0
Jul 31 19:42:36.079869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.079707    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"
Jul 31 19:42:36.079869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.079752    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" event={"ID":"8f60b341-8f40-45ec-9493-e9475116aa9a","Type":"ContainerDied","Data":"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"}
Jul 31 19:42:36.079869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.079799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4" event={"ID":"8f60b341-8f40-45ec-9493-e9475116aa9a","Type":"ContainerDied","Data":"0129c75a5ffc8655f172328b9f8c05b97c9c1a8f7889a5ae1522d49977977b27"}
Jul 31 19:42:36.079869 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.079822    2557 scope.go:117] "RemoveContainer" containerID="c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"
Jul 31 19:42:36.081032 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.081014    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" containerID="547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9" exitCode=0
Jul 31 19:42:36.081120 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.081052    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"
Jul 31 19:42:36.081120 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.081097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" event={"ID":"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449","Type":"ContainerDied","Data":"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"}
Jul 31 19:42:36.081241 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.081141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv" event={"ID":"6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449","Type":"ContainerDied","Data":"01acdbb1725273783a129c6e56fe3ee45a73f1f7c163121422f8063ab980f24b"}
Jul 31 19:42:36.082554 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.082530    2557 generic.go:356] "Generic (PLEG): container finished" podID="44607a0f-fafa-4797-8da7-383bd5aead89" containerID="819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea" exitCode=0
Jul 31 19:42:36.082646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.082563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" event={"ID":"44607a0f-fafa-4797-8da7-383bd5aead89","Type":"ContainerDied","Data":"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"}
Jul 31 19:42:36.082646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.082585    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"
Jul 31 19:42:36.082646 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.082617    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c" event={"ID":"44607a0f-fafa-4797-8da7-383bd5aead89","Type":"ContainerDied","Data":"beab3f6671e03d667af8554a2cb8f0448e380cd15328ebc86197d625f6d18d68"}
Jul 31 19:42:36.090878 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.090848    2557 scope.go:117] "RemoveContainer" containerID="c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"
Jul 31 19:42:36.091129 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:36.091110    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542\": container with ID starting with c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542 not found: ID does not exist" containerID="c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"
Jul 31 19:42:36.091205 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.091138    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542"} err="failed to get container status \"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542\": rpc error: code = NotFound desc = could not find container \"c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542\": container with ID starting with c30eb03831b9ed3931999b053e232004503d963e41a2eee6d974696fe0a39542 not found: ID does not exist"
Jul 31 19:42:36.091205 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.091159    2557 scope.go:117] "RemoveContainer" containerID="547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"
Jul 31 19:42:36.099402 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.099388    2557 scope.go:117] "RemoveContainer" containerID="547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"
Jul 31 19:42:36.099658 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:36.099641    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9\": container with ID starting with 547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9 not found: ID does not exist" containerID="547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"
Jul 31 19:42:36.099718 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.099665    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9"} err="failed to get container status \"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9\": rpc error: code = NotFound desc = could not find container \"547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9\": container with ID starting with 547661927f975d22ddaab80e05445e70bce31bcf5ced972abb7ba7ea2192a3b9 not found: ID does not exist"
Jul 31 19:42:36.099718 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.099682    2557 scope.go:117] "RemoveContainer" containerID="819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"
Jul 31 19:42:36.107093 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.107070    2557 scope.go:117] "RemoveContainer" containerID="819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"
Jul 31 19:42:36.107325 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:36.107306    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea\": container with ID starting with 819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea not found: ID does not exist" containerID="819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"
Jul 31 19:42:36.107384 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.107332    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea"} err="failed to get container status \"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea\": rpc error: code = NotFound desc = could not find container \"819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea\": container with ID starting with 819dc8f0db28e476f7565c7de57056eaaa74e0491b031314090e2e3a82c1b8ea not found: ID does not exist"
Jul 31 19:42:36.113115 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.113094    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:42:36.115551 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.115530    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd0284-7895f5db7d-5zbp4"]
Jul 31 19:42:36.129254 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.129234    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:42:36.138837 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.138814    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd0284-6d86f7cd5b-vdthv"]
Jul 31 19:42:36.157469 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.157429    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:42:36.159378 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.159359    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd0284-777cbc9895-l9x2c"]
Jul 31 19:42:36.183589 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.183566    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44607a0f-fafa-4797-8da7-383bd5aead89" path="/var/lib/kubelet/pods/44607a0f-fafa-4797-8da7-383bd5aead89/volumes"
Jul 31 19:42:36.183838 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.183826    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2843ff-7cdf-4e07-9d96-74172e23cae8" path="/var/lib/kubelet/pods/5f2843ff-7cdf-4e07-9d96-74172e23cae8/volumes"
Jul 31 19:42:36.184095 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.184083    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" path="/var/lib/kubelet/pods/6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449/volumes"
Jul 31 19:42:36.184380 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:36.184369    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f60b341-8f40-45ec-9493-e9475116aa9a" path="/var/lib/kubelet/pods/8f60b341-8f40-45ec-9493-e9475116aa9a/volumes"
Jul 31 19:42:42.698062 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.698029    2557 scope.go:117] "RemoveContainer" containerID="397d4850c2506e285ea30f564f2edd481b30a9d4dfd7812e022ee295c356a62c"
Jul 31 19:42:42.959560 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959480    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"]
Jul 31 19:42:42.959809 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959798    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44607a0f-fafa-4797-8da7-383bd5aead89" containerName="maas-api"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959811    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44607a0f-fafa-4797-8da7-383bd5aead89" containerName="maas-api"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959819    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" containerName="maas-api"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959824    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" containerName="maas-api"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959834    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" containerName="payload-pre-processing"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959839    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" containerName="payload-pre-processing"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959847    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f60b341-8f40-45ec-9493-e9475116aa9a" containerName="payload-processing"
Jul 31 19:42:42.959856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959853    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f60b341-8f40-45ec-9493-e9475116aa9a" containerName="payload-processing"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959864    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25672e9-d634-4653-a424-db451f303658" containerName="revoke-keys"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959870    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25672e9-d634-4653-a424-db451f303658" containerName="revoke-keys"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959916    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25672e9-d634-4653-a424-db451f303658" containerName="revoke-keys"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959922    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7f8b07-57c4-4b9d-a9ae-e880c2d1d449" containerName="payload-pre-processing"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959928    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f60b341-8f40-45ec-9493-e9475116aa9a" containerName="payload-processing"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959936    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="44607a0f-fafa-4797-8da7-383bd5aead89" containerName="maas-api"
Jul 31 19:42:42.960068 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.959941    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a23b83a-ac67-4fed-bf0c-ecb22f07bb49" containerName="maas-api"
Jul 31 19:42:42.964739 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.964723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:42.977175 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.977151    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4qkh\""
Jul 31 19:42:42.980402 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.980374    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"]
Jul 31 19:42:42.997331 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:42.997307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09a154-54jb8\" (UID: \"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:43.097880 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:43.097849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09a154-54jb8\" (UID: \"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:43.098726 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:43.098704    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09a154-54jb8\" (UID: \"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:43.274667 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:43.274585    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:43.429927 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:43.429902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"]
Jul 31 19:42:43.431487 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:42:43.431433    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d3d8b9_cc84_4632_97ab_ec30588a3cf2.slice/crio-af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327 WatchSource:0}: Error finding container af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327: Status 404 returned error can't find the container with id af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327
Jul 31 19:42:44.126020 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:44.125985    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8" event={"ID":"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2","Type":"ContainerStarted","Data":"228e512eebee10465fb8070af71f9509b38af5a03288107690e53819c14e614c"}
Jul 31 19:42:44.126483 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:44.126042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8" event={"ID":"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2","Type":"ContainerStarted","Data":"af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327"}
Jul 31 19:42:44.144953 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:44.144910    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8" podStartSLOduration=2.144894401 podStartE2EDuration="2.144894401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 19:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:42:44.14343423 +0000 UTC m=+3542.491016105" watchObservedRunningTime="2026-07-31 19:42:44.144894401 +0000 UTC m=+3542.492476273"
Jul 31 19:42:45.131779 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:45.131740    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" containerID="228e512eebee10465fb8070af71f9509b38af5a03288107690e53819c14e614c" exitCode=0
Jul 31 19:42:45.132188 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:45.131825    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8" event={"ID":"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2","Type":"ContainerDied","Data":"228e512eebee10465fb8070af71f9509b38af5a03288107690e53819c14e614c"}
Jul 31 19:42:46.268503 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:46.268479    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:46.325074 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:46.325036    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca\") pod \"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2\" (UID: \"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2\") "
Jul 31 19:42:46.325428 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:46.325400    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" (UID: "d0d3d8b9-cc84-4632-97ab-ec30588a3cf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:46.425608 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:46.425522    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2-maas-api-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:47.142457 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:47.142426    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"
Jul 31 19:42:47.142663 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:47.142469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8" event={"ID":"d0d3d8b9-cc84-4632-97ab-ec30588a3cf2","Type":"ContainerDied","Data":"af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327"}
Jul 31 19:42:47.142663 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:47.142510    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af22c25c1840eac31efd7560cadf80b96b12040f3b1ee7b5a06b14e593cf9327"
Jul 31 19:42:53.450080 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.450038    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:42:53.450800 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.450268    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" podUID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" containerName="maas-api" containerID="cri-o://046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a" gracePeriod=30
Jul 31 19:42:53.481490 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.481465    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"]
Jul 31 19:42:53.484918 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.484897    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-09a154-29758770-j9tfx"]
Jul 31 19:42:53.512522 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.512489    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:42:53.660408 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.660378    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"]
Jul 31 19:42:53.663359 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.663336    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09a154-54jb8"]
Jul 31 19:42:53.698529 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.698501    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:42:53.792694 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.792599    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca\") pod \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") "
Jul 31 19:42:53.792874 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.792705    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") pod \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") "
Jul 31 19:42:53.792874 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.792734    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xbd\" (UniqueName: \"kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd\") pod \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\" (UID: \"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848\") "
Jul 31 19:42:53.792984 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.792947    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" (UID: "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:53.794795 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.794773    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" (UID: "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 19:42:53.794885 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.794862    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd" (OuterVolumeSpecName: "kube-api-access-g9xbd") pod "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" (UID: "a3c5ea88-fb12-4b35-bff2-1bf2b8be4848"). InnerVolumeSpecName "kube-api-access-g9xbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:53.894247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.894204    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xbd\" (UniqueName: \"kubernetes.io/projected/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-kube-api-access-g9xbd\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:53.894247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.894239    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-openshift-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:53.894247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:53.894253    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848-maas-api-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:54.178540 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.178509    2557 generic.go:356] "Generic (PLEG): container finished" podID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" containerID="046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a" exitCode=0
Jul 31 19:42:54.178735 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.178633    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"
Jul 31 19:42:54.178928 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.178904    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" podUID="1761937a-1841-4541-9b38-d3c6d30d50b7" containerName="payload-processing" containerID="cri-o://b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6" gracePeriod=30
Jul 31 19:42:54.184247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.184225    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444fca39-3cde-4b8a-b68f-5a4110bb7daf" path="/var/lib/kubelet/pods/444fca39-3cde-4b8a-b68f-5a4110bb7daf/volumes"
Jul 31 19:42:54.184534 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.184519    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" path="/var/lib/kubelet/pods/d0d3d8b9-cc84-4632-97ab-ec30588a3cf2/volumes"
Jul 31 19:42:54.184854 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.184835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" event={"ID":"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848","Type":"ContainerDied","Data":"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"}
Jul 31 19:42:54.184940 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.184872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v" event={"ID":"a3c5ea88-fb12-4b35-bff2-1bf2b8be4848","Type":"ContainerDied","Data":"e7a1597394332e28d4accd41b6e3a81065381f4819b8de4e85cc5512bb46c912"}
Jul 31 19:42:54.184940 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.184895    2557 scope.go:117] "RemoveContainer" containerID="046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"
Jul 31 19:42:54.232323 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.232302    2557 scope.go:117] "RemoveContainer" containerID="046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"
Jul 31 19:42:54.232656 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:54.232632    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a\": container with ID starting with 046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a not found: ID does not exist" containerID="046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"
Jul 31 19:42:54.232744 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.232668    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a"} err="failed to get container status \"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a\": rpc error: code = NotFound desc = could not find container \"046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a\": container with ID starting with 046898a6fb36984517d1b3165cd150d59e66cc1d8b4139909509b4667835ed9a not found: ID does not exist"
Jul 31 19:42:54.243289 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.243265    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:42:54.249393 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.249370    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09a154-6cfc55f6b4-btf7v"]
Jul 31 19:42:54.415856 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.415830    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:42:54.499667 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.499636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnhs7\" (UniqueName: \"kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7\") pod \"1761937a-1841-4541-9b38-d3c6d30d50b7\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") "
Jul 31 19:42:54.500061 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.499712    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume\") pod \"1761937a-1841-4541-9b38-d3c6d30d50b7\" (UID: \"1761937a-1841-4541-9b38-d3c6d30d50b7\") "
Jul 31 19:42:54.500108 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.500070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1761937a-1841-4541-9b38-d3c6d30d50b7" (UID: "1761937a-1841-4541-9b38-d3c6d30d50b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 19:42:54.501844 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.501826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7" (OuterVolumeSpecName: "kube-api-access-fnhs7") pod "1761937a-1841-4541-9b38-d3c6d30d50b7" (UID: "1761937a-1841-4541-9b38-d3c6d30d50b7"). InnerVolumeSpecName "kube-api-access-fnhs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 19:42:54.600550 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.600513    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnhs7\" (UniqueName: \"kubernetes.io/projected/1761937a-1841-4541-9b38-d3c6d30d50b7-kube-api-access-fnhs7\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:54.600550 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:54.600545    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1761937a-1841-4541-9b38-d3c6d30d50b7-plugins-config-volume\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
Jul 31 19:42:55.184391 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.184363    2557 generic.go:356] "Generic (PLEG): container finished" podID="1761937a-1841-4541-9b38-d3c6d30d50b7" containerID="b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6" exitCode=0
Jul 31 19:42:55.184634 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.184408    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"
Jul 31 19:42:55.184634 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.184470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" event={"ID":"1761937a-1841-4541-9b38-d3c6d30d50b7","Type":"ContainerDied","Data":"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"}
Jul 31 19:42:55.184634 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.184519    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68" event={"ID":"1761937a-1841-4541-9b38-d3c6d30d50b7","Type":"ContainerDied","Data":"54d32c9c7a1a75795311ed28010a573d00bd469d20f6c4bf52b0cc16a2ba11d6"}
Jul 31 19:42:55.184634 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.184542    2557 scope.go:117] "RemoveContainer" containerID="b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"
Jul 31 19:42:55.193536 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.193508    2557 scope.go:117] "RemoveContainer" containerID="b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"
Jul 31 19:42:55.193749 ip-10-0-128-163 kubenswrapper[2557]: E0731 19:42:55.193732    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6\": container with ID starting with b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6 not found: ID does not exist" containerID="b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"
Jul 31 19:42:55.193792 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.193757    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6"} err="failed to get container status \"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6\": rpc error: code = NotFound desc = could not find container \"b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6\": container with ID starting with b4e6c94b27e1806016e066293bfb372ef5f0efb795dbd069012416537c3a96b6 not found: ID does not exist"
Jul 31 19:42:55.209080 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.209054    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:42:55.210792 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:55.210769    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-09a154-58b966dd98-gxq68"]
Jul 31 19:42:56.188618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:56.183914    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1761937a-1841-4541-9b38-d3c6d30d50b7" path="/var/lib/kubelet/pods/1761937a-1841-4541-9b38-d3c6d30d50b7/volumes"
Jul 31 19:42:56.188618 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:42:56.184552    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" path="/var/lib/kubelet/pods/a3c5ea88-fb12-4b35-bff2-1bf2b8be4848/volumes"
Jul 31 19:43:06.894120 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:06.894091    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-vm4jm_e7b6e52e-ff8b-4c80-8d30-524ca2e6707e/copy-manifests/0.log"
Jul 31 19:43:06.907824 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:06.907794    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-vm4jm_e7b6e52e-ff8b-4c80-8d30-524ca2e6707e/manager/0.log"
Jul 31 19:43:07.581672 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:07.581639    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758755-xfs6g_0c61f32b-9cc3-459e-98e2-f125be421cb7/cleanup/0.log"
Jul 31 19:43:09.188590 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:09.188554    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-kftfz_0cae623c-0f33-4820-8917-a68ffd41eeb1/manager/0.log"
Jul 31 19:43:09.637018 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:09.636983    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-88mdk_c164c810-acc7-48b5-b306-f7d1ceec06fd/discovery/0.log"
Jul 31 19:43:10.640876 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:10.640844    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-m86zj_7d93fcb4-dd4d-4628-827e-84182484c5ce/main/0.log"
Jul 31 19:43:11.067364 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:11.067289    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-d5825_2d0ad837-6657-47bf-9fd7-789210d97331/main/0.log"
Jul 31 19:43:19.557129 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:19.557096    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fhf8w_df4356d7-969b-4b93-a3e6-3fb45690cd29/global-pull-secret-syncer/0.log"
Jul 31 19:43:19.650631 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:19.650604    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-99v4g_89e00fd6-040c-4723-b241-289e17f0ef8e/konnectivity-agent/0.log"
Jul 31 19:43:19.723642 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:19.723596    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-163.ec2.internal_fda49057b793c5c62334b176ef9bc360/haproxy/0.log"
Jul 31 19:43:25.280082 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:25.280042    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-kftfz_0cae623c-0f33-4820-8917-a68ffd41eeb1/manager/0.log"
Jul 31 19:43:27.178229 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:27.178190    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29kkd_706d6d31-2fab-47d2-a1f2-36e95622be06/node-exporter/0.log"
Jul 31 19:43:27.203909 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:27.203885    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29kkd_706d6d31-2fab-47d2-a1f2-36e95622be06/kube-rbac-proxy/0.log"
Jul 31 19:43:27.228906 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:27.228879    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29kkd_706d6d31-2fab-47d2-a1f2-36e95622be06/init-textfile/0.log"
Jul 31 19:43:29.532997 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.532961    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"]
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533255    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1761937a-1841-4541-9b38-d3c6d30d50b7" containerName="payload-processing"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533266    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1761937a-1841-4541-9b38-d3c6d30d50b7" containerName="payload-processing"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533276    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" containerName="revoke-keys"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533281    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" containerName="revoke-keys"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533298    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" containerName="maas-api"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533303    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" containerName="maas-api"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533356    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1761937a-1841-4541-9b38-d3c6d30d50b7" containerName="payload-processing"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533362    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d3d8b9-cc84-4632-97ab-ec30588a3cf2" containerName="revoke-keys"
Jul 31 19:43:29.533375 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.533369    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c5ea88-fb12-4b35-bff2-1bf2b8be4848" containerName="maas-api"
Jul 31 19:43:29.536516 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.536493    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.539373 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.539354    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"kube-root-ca.crt\""
Jul 31 19:43:29.539534 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.539361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x2d7j\"/\"default-dockercfg-kfhf6\""
Jul 31 19:43:29.540489 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.540471    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x2d7j\"/\"openshift-service-ca.crt\""
Jul 31 19:43:29.545701 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.545681    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"]
Jul 31 19:43:29.588425 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.588398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-proc\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.588425 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.588428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hlf\" (UniqueName: \"kubernetes.io/projected/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-kube-api-access-b4hlf\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.588638 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.588484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-podres\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.588638 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.588519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-lib-modules\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.588638 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.588543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-sys\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689742 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-podres\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-lib-modules\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-sys\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689869    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-sys\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689887    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-podres\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-proc\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689921    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-proc\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.689941 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-lib-modules\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.690288 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.689940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hlf\" (UniqueName: \"kubernetes.io/projected/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-kube-api-access-b4hlf\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.699454 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.699408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hlf\" (UniqueName: \"kubernetes.io/projected/0c2a35f3-1de3-4570-8503-4bfcf25de3c3-kube-api-access-b4hlf\") pod \"perf-node-gather-daemonset-h5rtz\" (UID: \"0c2a35f3-1de3-4570-8503-4bfcf25de3c3\") " pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.847081 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.846995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:29.889753 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.889727    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:43:29.895147 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.895121    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/2.log"
Jul 31 19:43:29.980341 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:29.980316    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"]
Jul 31 19:43:29.982318 ip-10-0-128-163 kubenswrapper[2557]: W0731 19:43:29.982290    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c2a35f3_1de3_4570_8503_4bfcf25de3c3.slice/crio-a81cd2d6ed521e5a099bd709c0cdf792caf407ac61bd6084cb57d5496d4f602c WatchSource:0}: Error finding container a81cd2d6ed521e5a099bd709c0cdf792caf407ac61bd6084cb57d5496d4f602c: Status 404 returned error can't find the container with id a81cd2d6ed521e5a099bd709c0cdf792caf407ac61bd6084cb57d5496d4f602c
Jul 31 19:43:30.360737 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:30.360696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz" event={"ID":"0c2a35f3-1de3-4570-8503-4bfcf25de3c3","Type":"ContainerStarted","Data":"3c9601d30125ab119a4426f251b6f3bd0470530e03753bead795a5ef4c925d09"}
Jul 31 19:43:30.360951 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:30.360748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz" event={"ID":"0c2a35f3-1de3-4570-8503-4bfcf25de3c3","Type":"ContainerStarted","Data":"a81cd2d6ed521e5a099bd709c0cdf792caf407ac61bd6084cb57d5496d4f602c"}
Jul 31 19:43:30.360951 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:30.360806    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:30.378247 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:30.378198    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz" podStartSLOduration=1.3781842 podStartE2EDuration="1.3781842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 19:43:30.377436224 +0000 UTC m=+3588.725018107" watchObservedRunningTime="2026-07-31 19:43:30.3781842 +0000 UTC m=+3588.725766080"
Jul 31 19:43:31.222007 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:31.221977    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-g5mf5_7031db7a-a3ee-45f4-8448-4913e8a110c9/volume-data-source-validator/0.log"
Jul 31 19:43:32.408540 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:32.408501    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ps6l_5a0497e7-04d8-4120-81fe-23357e4d9b1b/dns/0.log"
Jul 31 19:43:32.433359 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:32.433331    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ps6l_5a0497e7-04d8-4120-81fe-23357e4d9b1b/kube-rbac-proxy/0.log"
Jul 31 19:43:32.565805 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:32.565778    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rprxx_f7ed081d-85b1-41da-af82-2caff6b85486/dns-node-resolver/0.log"
Jul 31 19:43:33.157451 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:33.157414    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d66d8_9f8de46e-3b7b-4896-a7c4-2256edda4630/node-ca/0.log"
Jul 31 19:43:34.355198 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:34.355166    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-88mdk_c164c810-acc7-48b5-b306-f7d1ceec06fd/discovery/0.log"
Jul 31 19:43:35.391152 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:35.391125    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2nvps_8ceefe50-afa0-49e4-ab29-a898d0173807/serve-healthcheck-canary/0.log"
Jul 31 19:43:36.063977 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:36.063938    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rnq2_ca122841-d193-49fa-94e7-e3de5335fe03/kube-rbac-proxy/0.log"
Jul 31 19:43:36.090850 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:36.090826    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rnq2_ca122841-d193-49fa-94e7-e3de5335fe03/exporter/0.log"
Jul 31 19:43:36.119413 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:36.119386    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rnq2_ca122841-d193-49fa-94e7-e3de5335fe03/extractor/0.log"
Jul 31 19:43:36.375978 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:36.375946    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x2d7j/perf-node-gather-daemonset-h5rtz"
Jul 31 19:43:39.225334 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:39.225295    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-vm4jm_e7b6e52e-ff8b-4c80-8d30-524ca2e6707e/manager/0.log"
Jul 31 19:43:39.249735 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:39.249709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84c85c7ff9-vm4jm_e7b6e52e-ff8b-4c80-8d30-524ca2e6707e/copy-manifests/0.log"
Jul 31 19:43:40.187240 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:40.187217    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-88mdk_c164c810-acc7-48b5-b306-f7d1ceec06fd/discovery/0.log"
Jul 31 19:43:40.901348 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:40.901319    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6478f7cf66-pb29v_a6ae8e0a-d99d-446d-8cf8-d3822544c1d5/manager/0.log"
Jul 31 19:43:41.973672 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:41.973644    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-88mdk_c164c810-acc7-48b5-b306-f7d1ceec06fd/discovery/0.log"
Jul 31 19:43:42.435800 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:42.435702    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:43:42.453790 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:42.443040    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-b2gjc_7da0140a-41df-45f3-bbd3-0cc2d319a729/console-operator/1.log"
Jul 31 19:43:42.735305 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:42.735276    2557 scope.go:117] "RemoveContainer" containerID="595eeca6571484650ed2fc39a56a28429a816d76e38057ed5e42cdd4617d6728"
Jul 31 19:43:46.102046 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:46.102016    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-l6rw2_6ddd4be7-e7ba-4f7c-b037-5a7756647d46/migrator/0.log"
Jul 31 19:43:46.127205 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:46.127157    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-l6rw2_6ddd4be7-e7ba-4f7c-b037-5a7756647d46/graceful-termination/0.log"
Jul 31 19:43:46.523642 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:46.523606    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-lqgxh_d0353f16-2658-49f0-b4de-87d3bcf0d529/kube-storage-version-migrator-operator/1.log"
Jul 31 19:43:46.524384 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:46.524364    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-lqgxh_d0353f16-2658-49f0-b4de-87d3bcf0d529/kube-storage-version-migrator-operator/0.log"
Jul 31 19:43:47.579047 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.579017    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4vcn4_06af2d8b-c23f-4b50-9cc4-5da5cc5b3880/kube-multus/0.log"
Jul 31 19:43:47.631503 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.631468    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/kube-multus-additional-cni-plugins/0.log"
Jul 31 19:43:47.688746 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.688715    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/egress-router-binary-copy/0.log"
Jul 31 19:43:47.729746 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.729717    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/cni-plugins/0.log"
Jul 31 19:43:47.758809 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.758786    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/bond-cni-plugin/0.log"
Jul 31 19:43:47.786824 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.786798    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/routeoverride-cni/0.log"
Jul 31 19:43:47.812497 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.812465    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/whereabouts-cni-bincopy/0.log"
Jul 31 19:43:47.837875 ip-10-0-128-163 kubenswrapper[2557]: I0731 19:43:47.837795    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-68swk_4505fee6-5b31-49a9-8125-2c57e9ae7be4/whereabouts-cni/0.log"