Jul 29 21:03:32.592003 ip-10-0-143-168 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 21:03:32.952885 ip-10-0-143-168 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:03:32.952885 ip-10-0-143-168 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 21:03:32.952885 ip-10-0-143-168 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:03:32.952885 ip-10-0-143-168 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 21:03:32.952885 ip-10-0-143-168 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:03:32.955157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.955057    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 21:03:32.960903 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960878    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.960903 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960902    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960908    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960912    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960915    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960918    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960921    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960924    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960927    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960930    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960932    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960935    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960937    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960940    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960943    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960946    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960953    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960956    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960958    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960961    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960964    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.960986 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960967    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960969    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960972    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960975    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960977    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960980    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960982    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960985    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960988    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960990    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960993    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960996    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.960999    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961002    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961005    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961008    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961010    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961013    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961015    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961018    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.961457 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961020    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961022    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961025    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961029    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961031    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961034    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961038    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961040    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961043    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961045    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961047    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961051    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961054    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961056    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961059    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961062    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961065    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961067    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961070    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961073    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.961976 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961075    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961079    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961081    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961084    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961086    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961089    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961091    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961094    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961096    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961099    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961104    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961108    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961111    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961114    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961117    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961119    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961122    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961126    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961129    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.962506 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961132    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961135    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961138    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961140    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961143    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961146    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961148    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961151    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961155    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961157    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961161    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961163    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961166    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961168    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961171    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961175    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961178    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961181    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961184    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961935    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.962980 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961943    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961946    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961949    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961952    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961959    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961963    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961966    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961969    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961972    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961974    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961977    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961979    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961982    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961985    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961988    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961990    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961994    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961996    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.961999    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962003    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.963463 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962005    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962008    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962012    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962014    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962017    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962019    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962023    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962026    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962029    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962031    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962035    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962039    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962042    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962045    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962048    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962051    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962054    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962056    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962059    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962062    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.963977 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962064    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962066    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962069    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962071    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962074    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962077    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962080    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962082    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962085    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962088    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962091    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962094    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962096    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962099    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962102    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962105    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962108    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962110    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962113    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962115    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.964466 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962118    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962120    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962123    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962126    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962128    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962131    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962136    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962139    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962142    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962145    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962148    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962150    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962153    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962156    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962158    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962163    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962166    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962169    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.964963 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962172    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962176    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962179    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962182    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962185    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962188    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962190    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962193    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962196    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962198    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962201    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962203    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962206    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962208    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962211    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962213    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962216    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962219    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962221    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.965446 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962606    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962611    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962614    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962617    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962620    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962623    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962626    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962628    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962631    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962633    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962636    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962640    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962643    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962647    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962650    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962653    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962656    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962659    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962662    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962665    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.966021 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962667    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962670    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962673    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962676    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962678    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962682    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962684    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962687    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962689    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962707    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962711    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962713    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962717    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962721    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962724    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962727    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962730    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962734    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962737    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.966507 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962741    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962744    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962746    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962749    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962752    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962754    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962757    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962760    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962763    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962770    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962773    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962775    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962778    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962781    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962783    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962786    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962789    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962791    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962794    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962796    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.966984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962799    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962801    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962805    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962808    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962810    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962813    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962815    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962818    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962821    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962823    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962826    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962830    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962833    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962836    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962838    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962841    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962843    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962846    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962848    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962851    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.967504 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962854    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962862    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962865    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962868    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962870    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962873    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962876    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962878    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962881    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962884    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962886    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962889    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962892    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962894    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962897    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962900    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962905    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962908    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962910    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.962913    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.968007 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963038    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963052    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963059    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963064    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963069    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963072    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963076    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963081    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963084    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963087    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963090    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963093    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963096    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963101    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963104    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963107    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963110    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963113    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963117    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963120    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963123    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963127    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963130    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963136    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 21:03:32.968478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963140    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963143    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963146    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963150    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963153    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963156    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963159    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963162    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963166    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963169    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963173    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963175    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963179    2581 flags.go:64] FLAG: --enable-server="true"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963182    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963187    2581 flags.go:64] FLAG: --event-burst="100"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963190    2581 flags.go:64] FLAG: --event-qps="50"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963193    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963196    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963199    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963203    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963206    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963210    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963214    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963216    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963219    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 21:03:32.969078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963222    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963226    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963229    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963232    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963235    2581 flags.go:64] FLAG: --feature-gates=""
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963239    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963243    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963246    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963249    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963253    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963256    2581 flags.go:64] FLAG: --help="false"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963259    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-168.ec2.internal"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963262    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963275    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963278    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963282    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963286    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963289    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963292    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963295    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963298    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963301    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963305    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963308    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 29 21:03:32.969738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963311    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963314    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963317    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963320    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963324    2581 flags.go:64] FLAG: --lock-file=""
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963327    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963330    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963333    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963339    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963342    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963345    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963348    2581 flags.go:64] FLAG: --logging-format="text"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963351    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963354    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963360    2581 flags.go:64] FLAG: --manifest-url=""
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963363    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963368    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963371    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963375    2581 flags.go:64] FLAG: --max-pods="110"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963378    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963381    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963384    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963387    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963390    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963393    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 21:03:32.970340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963396    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963405    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963408    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963411    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963414    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963418    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963424    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963427    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963431    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963434    2581 flags.go:64] FLAG: --port="10250"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963436    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963441    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9bab94101c4b9cd"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963444    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963447    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963450    2581 flags.go:64] FLAG: --register-node="true"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963453    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963457    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963460    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963462    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963465    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963469    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963473    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963477    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963480    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963483    2581 flags.go:64] FLAG: --runonce="false"
Jul 29 21:03:32.970996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963486    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963489    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963492    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963495    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963498    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963501    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963504    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963507    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963510    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963512    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963515    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963518    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963521    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963525    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963528    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963533    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963536    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963539    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963544    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963547    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963550    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963553    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963555    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963558    2581 flags.go:64] FLAG: --v="2"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963562    2581 flags.go:64] FLAG: --version="false"
Jul 29 21:03:32.971623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963567    2581 flags.go:64] FLAG: --vmodule=""
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963571    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.963574    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963711    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963719    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963722    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963725    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963728    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963730    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963733    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963736    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963738    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963741    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963744    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963747    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963749    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963752    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963755    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963757    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963760    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.972376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963762    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963786    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963789    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963791    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963794    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963798    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963801    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963803    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963806    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963808    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963811    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963813    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963815    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963818    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963821    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963823    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963827    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963830    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963832    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963835    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.972895 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963837    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963840    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963843    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963845    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963848    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963850    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963853    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963855    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963857    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963860    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963862    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963865    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963868    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963870    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963874    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963876    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963879    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963883    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963887    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963891    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.973392 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963894    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963897    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963900    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963902    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963905    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963907    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963910    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963913    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963917    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963920    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963923    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963925    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963928    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963930    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963933    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963936    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963938    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963941    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963943    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.973907 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963946    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963948    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963951    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963954    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963956    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963959    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963961    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963964    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963968    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963970    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963974    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963977    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963979    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963982    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963984    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963988    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963992    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963996    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.963999    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.964002    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.974367 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.964005    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.964010    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.964012    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.964018    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.970891    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.970910    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.971517    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.971526    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971589    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971595    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971599    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971602    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971605    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971608    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971611    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.974870 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971614    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971617    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971620    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971623    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971626    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971628    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971631    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971634    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971637    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971639    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971642    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971644    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971647    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971650    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971654    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971658    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971661    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971664    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971667    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.975275 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971669    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971672    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971676    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971681    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971685    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971688    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971691    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971714    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971717    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971720    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971723    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971725    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971730    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971732    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971735    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971738    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971740    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971743    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971745    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971748    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.975760 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971750    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971753    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971757    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971760    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971763    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971766    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971769    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971771    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971774    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971776    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971779    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971782    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971785    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971787    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971791    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971794    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971797    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971800    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971802    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971805    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.976245 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971809    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971811    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971814    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971817    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971819    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971822    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971824    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971827    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971830    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971832    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971835    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971838    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971840    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971843    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971845    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971848    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971850    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971853    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971855    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.976832 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971858    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971860    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971863    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971866    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971868    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971870    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971873    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971876    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971879    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971882    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971885    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971887    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971890    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.971892    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.971898    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:03:32.977289 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972039    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972045    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972048    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972051    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972054    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972057    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972059    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972062    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972064    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972067    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972069    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972073    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972077    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972080    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972082    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972085    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972087    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972089    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972092    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:03:32.977667 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972094    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972097    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972100    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972102    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972105    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972108    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972111    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972114    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972117    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972120    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972123    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972125    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972128    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972130    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972134    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972137    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972140    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972143    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972146    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:03:32.978135 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972148    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972151    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972153    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972156    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972158    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972161    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972163    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972166    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972168    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972171    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972173    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972176    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972178    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972181    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972183    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972186    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972188    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972191    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972194    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972196    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:03:32.978586 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972199    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972202    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972205    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972208    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972212    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972216    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972218    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972221    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972224    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972226    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972229    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972231    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972233    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972236    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972238    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972241    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972243    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972246    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972248    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:03:32.979128 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972251    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972253    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972256    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972258    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972261    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972263    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972266    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972269    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972271    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972273    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972276    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972279    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972281    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972284    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972287    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972289    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972292    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972295    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972297    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972299    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:03:32.979610 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972302    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:32.972305    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.972309    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.972444    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.974816    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.976165    2581 server.go:1019] "Starting client certificate rotation"
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.976272    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.976341    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 21:03:32.980150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.978774    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 21:03:32.985657 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.985631    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 21:03:32.995784 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.995756    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 29 21:03:32.997911 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.997894    2581 log.go:25] "Validated CRI v1 image API"
Jul 29 21:03:32.998801 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:32.998783    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 21:03:33.001304 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.001281    2581 fs.go:135] Filesystem UUIDs: map[164ac904-50af-4dd6-8616-8c525fdbb30a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fa2ca84c-0eab-41a5-8350-26366c81f733:/dev/nvme0n1p3]
Jul 29 21:03:33.001358 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.001304    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 21:03:33.006274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.006155    2581 manager.go:216] Machine: {Timestamp:2026-07-29 21:03:33.005067084 +0000 UTC m=+0.303544830 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105704 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0a69e3ed4ee7238e0cccd38fef78 SystemUUID:ec2f0a69-e3ed-4ee7-238e-0cccd38fef78 BootID:68e0b586-eacb-4089-b8a9-f55ae5f07b3e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a2:7e:03:57:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a2:7e:03:57:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:a4:d6:cf:71:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 21:03:33.006274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.006269    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 21:03:33.006384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.006361    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 21:03:33.007480 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.007436    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 21:03:33.007635 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.007483    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 21:03:33.008747 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.008735    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 21:03:33.008780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.008749    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 21:03:33.008780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.008757    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 21:03:33.009336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.009326    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 21:03:33.010124 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.010112    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 21:03:33.010170 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.010155    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 21:03:33.011800 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.011787    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 21:03:33.011929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.011919    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 21:03:33.012516 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012503    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 21:03:33.012556 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012536    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 21:03:33.012584 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012556    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 21:03:33.012584 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012567    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 29 21:03:33.012584 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012577    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 21:03:33.013002 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.012988    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 21:03:33.013038 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.013017    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 21:03:33.015625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.015610    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 21:03:33.017730 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.017716    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 21:03:33.017772 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.017749    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 21:03:33.019400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019386    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019406    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019415    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019421    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019430    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019435    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 21:03:33.019451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019446    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 21:03:33.019602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019455    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 21:03:33.019602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019466    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 21:03:33.019602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019474    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 21:03:33.019602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019496    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 21:03:33.019602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.019505    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 21:03:33.020151 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.020137    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 21:03:33.020187 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.020154    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 21:03:33.022954 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.022937    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 21:03:33.023025 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.022945    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 21:03:33.023025 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.022945    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 21:03:33.023709 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.023684    2581 server.go:1268] "Started kubelet"
Jul 29 21:03:33.024062 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.023771    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 21:03:33.024062 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.023831    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 21:03:33.024062 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.023916    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 21:03:33.024839 ip-10-0-143-168 systemd[1]: Started Kubernetes Kubelet.
Jul 29 21:03:33.028780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.028750    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 21:03:33.029542 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.029516    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 21:03:33.030617 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.030226    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 21:03:33.032649 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.032632    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lf4sf"
Jul 29 21:03:33.035064 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035042    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 21:03:33.035158 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035045    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 21:03:33.035228 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.035153    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.035228 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.034247    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-168.ec2.internal.18c6de9027d07a4e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-168.ec2.internal,UID:ip-10-0-143-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-168.ec2.internal,},FirstTimestamp:2026-07-29 21:03:33.02361147 +0000 UTC m=+0.322089216,LastTimestamp:2026-07-29 21:03:33.02361147 +0000 UTC m=+0.322089216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-168.ec2.internal,}"
Jul 29 21:03:33.035615 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035597    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 21:03:33.035615 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035616    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 21:03:33.035774 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035599    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 21:03:33.035774 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035763    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 21:03:33.035774 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.035773    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 21:03:33.036281 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036262    2581 factory.go:153] Registering CRI-O factory
Jul 29 21:03:33.036281 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036283    2581 factory.go:223] Registration of the crio container factory successfully
Jul 29 21:03:33.036405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036340    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 21:03:33.036405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036352    2581 factory.go:55] Registering systemd factory
Jul 29 21:03:33.036405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036360    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 29 21:03:33.036405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036385    2581 factory.go:103] Registering Raw factory
Jul 29 21:03:33.036405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036399    2581 manager.go:1196] Started watching for new ooms in manager
Jul 29 21:03:33.036899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.036884    2581 manager.go:349] Starting recovery of all containers
Jul 29 21:03:33.039387 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.039354    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 21:03:33.039387 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.039370    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 21:03:33.040123 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.040098    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lf4sf"
Jul 29 21:03:33.048218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.048033    2581 manager.go:354] Recovery completed
Jul 29 21:03:33.053149 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.053136    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.055238 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055219    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.055320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055251    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.055320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055275    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.055930 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055916    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 21:03:33.055980 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055930    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 21:03:33.055980 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055946    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 21:03:33.056049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.055989    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.056854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.056839    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.056922 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.056872    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.056922 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.056884    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.057509 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.057499    2581 policy_none.go:49] "None policy: Start"
Jul 29 21:03:33.057544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.057519    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 21:03:33.057544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.057530    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 21:03:33.057619 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.057543    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-168.ec2.internal.18c6de9029b30ddd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-168.ec2.internal,UID:ip-10-0-143-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-168.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-168.ec2.internal,},FirstTimestamp:2026-07-29 21:03:33.055237597 +0000 UTC m=+0.353715342,LastTimestamp:2026-07-29 21:03:33.055237597 +0000 UTC m=+0.353715342,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-168.ec2.internal,}"
Jul 29 21:03:33.058441 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.058431    2581 policy_none.go:47] "Start"
Jul 29 21:03:33.108159 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108142    2581 manager.go:337] "Starting Device Plugin manager"
Jul 29 21:03:33.108233 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.108189    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 21:03:33.108233 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108203    2581 server.go:87] "Starting device plugin registration server"
Jul 29 21:03:33.108478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108466    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 21:03:33.108577 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108479    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 21:03:33.108627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108592    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 21:03:33.108682 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108672    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 21:03:33.108743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.108683    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 21:03:33.109310 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.109286    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 21:03:33.109410 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.109337    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.179092 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.179050    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 21:03:33.181312 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.180431    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 21:03:33.181312 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.180469    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 21:03:33.181312 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.180496    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 21:03:33.181312 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.180539    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 21:03:33.183140 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.183116    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:33.209366 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.209286    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.210413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.210395    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.210532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.210432    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.210532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.210447    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.210532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.210486    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.218652 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.218626    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.218766 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.218656    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-168.ec2.internal\": node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.235274 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.235233    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.281331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.281292    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"]
Jul 29 21:03:33.281496 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.281394    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.283149 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.283132    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.283206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.283167    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.283206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.283179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.285501 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.285489    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.285608 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.285591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.285644 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.285630    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.286467 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.286445    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.286576 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.286477    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.286576 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.286489    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.287521 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.287504    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.287635 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.287536    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.287635 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.287551    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.288750 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.288731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.288852 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.288777    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:03:33.289568 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.289555    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:03:33.289650 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.289580    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:03:33.289650 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.289589    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:03:33.314663 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.314638    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-168.ec2.internal\" not found" node="ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.319240 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.319219    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-168.ec2.internal\" not found" node="ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.335436 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.335398    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.435973 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.435943    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.437085 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.437066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.437151 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.437094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df72579e9d2f6ec664c78ab551d5adbe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"df72579e9d2f6ec664c78ab551d5adbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.437151 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.437114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.536594 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.536518    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.537672 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.537768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.537768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df72579e9d2f6ec664c78ab551d5adbe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"df72579e9d2f6ec664c78ab551d5adbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.537861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.537861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df72579e9d2f6ec664c78ab551d5adbe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-168.ec2.internal\" (UID: \"df72579e9d2f6ec664c78ab551d5adbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.537861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.537827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c807facfd39d2d720088d43e7e67d0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal\" (UID: \"2c807facfd39d2d720088d43e7e67d0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.617894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.617842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.622985 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.622760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:33.637380 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.637353    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.737869 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.737817    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.838370 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.838277    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.916190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.916165    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:33.939193 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:33.939160    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:33.977322 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.977293    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 21:03:33.977871 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.977483    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 21:03:33.977871 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:33.977508    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 21:03:34.039463 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:34.039423    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:34.044130 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.044101    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:58:33 +0000 UTC" deadline="2028-02-22 07:47:39.430948642 +0000 UTC"
Jul 29 21:03:34.044130 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.044129    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13738h44m5.386823282s"
Jul 29 21:03:34.139648 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:34.139573    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:34.239923 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:34.239884    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:34.295214 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:34.295174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c807facfd39d2d720088d43e7e67d0a.slice/crio-de256c81d11fa0075099e5188f2e2236c4228281e1e3f43653db509a80193bca WatchSource:0}: Error finding container de256c81d11fa0075099e5188f2e2236c4228281e1e3f43653db509a80193bca: Status 404 returned error can't find the container with id de256c81d11fa0075099e5188f2e2236c4228281e1e3f43653db509a80193bca
Jul 29 21:03:34.299565 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.299544    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:03:34.340663 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:34.340628    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-168.ec2.internal\" not found"
Jul 29 21:03:34.353010 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.352986    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:34.441943 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.441843    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:34.454648 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.454614    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 21:03:34.455738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.455721    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal"
Jul 29 21:03:34.466104 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.466072    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 21:03:34.561173 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:34.561138    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:34.862276 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:34.862241    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf72579e9d2f6ec664c78ab551d5adbe.slice/crio-bf18f753c70fa422a5891923366051860583c1bbcf45de6d7dfc6cac98fa2325 WatchSource:0}: Error finding container bf18f753c70fa422a5891923366051860583c1bbcf45de6d7dfc6cac98fa2325: Status 404 returned error can't find the container with id bf18f753c70fa422a5891923366051860583c1bbcf45de6d7dfc6cac98fa2325
Jul 29 21:03:35.013990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.013956    2581 apiserver.go:52] "Watching apiserver"
Jul 29 21:03:35.023765 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.023719    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 21:03:35.024820 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.024793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dn9wv","openshift-multus/multus-additional-cni-plugins-mk8nn","openshift-multus/network-metrics-daemon-4tv28","kube-system/konnectivity-agent-8vmkk","openshift-image-registry/node-ca-t9785","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal","openshift-multus/multus-g4xnp","openshift-network-diagnostics/network-check-target-rtcfx","openshift-network-operator/iptables-alerter-xhv5l","openshift-ovn-kubernetes/ovnkube-node-2xwnk","kube-system/global-pull-secret-syncer-t4q5p","kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f","openshift-cluster-node-tuning-operator/tuned-z8hth"]
Jul 29 21:03:35.027911 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.027886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:35.028022 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.027984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:35.030387 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.030363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.032669 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.032639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.032880 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.032862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.032955 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.032894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-85464\""
Jul 29 21:03:35.032955 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.032863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.032955 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.032919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 21:03:35.035041 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.035016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.035418 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.035392    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rv76t\""
Jul 29 21:03:35.035529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.035476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.036324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.035658    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 21:03:35.036513 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.036488    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 21:03:35.036612 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.036516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 21:03:35.036612 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.036517    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.037614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.037590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 21:03:35.037877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.037854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 21:03:35.038074 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.038060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hbbm9\""
Jul 29 21:03:35.038623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.038395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.040485 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.040467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.042447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.042193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 21:03:35.042447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.042318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ls92z\""
Jul 29 21:03:35.042447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.042344    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.042447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.042387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.043431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.043081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.043431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.043126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c9k4s\""
Jul 29 21:03:35.043431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.043135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.043431 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.043150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:35.043431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.043300    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.046453 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.045909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.046453 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-etc-kubernetes\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-netns\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:35.046621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-cnibin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-cni-binary-copy\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-k8s-cni-cncf-io\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-kubelet\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-bin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.046836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrdm\" (UniqueName: \"kubernetes.io/projected/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-kube-api-access-mbrdm\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.046836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ndl\" (UniqueName: \"kubernetes.io/projected/a6e6382c-dc81-4584-ac12-bd4a147c874f-kube-api-access-p7ndl\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de8b18af-2228-40dc-b006-01407c1c8ba7-konnectivity-ca\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-os-release\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-socket-dir-parent\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-conf-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-daemon-config\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.046981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-host\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.047054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-serviceca\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6e6382c-dc81-4584-ac12-bd4a147c874f-iptables-alerter-script\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6e6382c-dc81-4584-ac12-bd4a147c874f-host-slash\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-multus\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pz9\" (UniqueName: \"kubernetes.io/projected/23e94a11-f3fc-4b84-b714-3667c90e23c7-kube-api-access-l6pz9\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de8b18af-2228-40dc-b006-01407c1c8ba7-agent-certs\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-system-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-hostroot\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.047324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.047277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-multus-certs\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.048568 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.048546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.049129 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.049063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 21:03:35.049266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.049064    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s88d5\""
Jul 29 21:03:35.049447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.049427    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 21:03:35.049536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.049504    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 21:03:35.051152 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.051131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.051265 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.051210    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:35.051886 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.051867    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 21:03:35.052098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 21:03:35.052098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052091    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 21:03:35.052210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.052210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052133    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.052210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5z9r9\""
Jul 29 21:03:35.052210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.052199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 21:03:35.053960 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.053941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.059278 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.058848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.059417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.059307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 21:03:35.059417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.059315    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xd859\""
Jul 29 21:03:35.059417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.059357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.059417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.059364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.061574 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.061499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 21:03:35.061574 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.061547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:03:35.061782 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.061553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2xbt\""
Jul 29 21:03:35.072070 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.072044    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ssq9x"
Jul 29 21:03:35.081328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.081305    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ssq9x"
Jul 29 21:03:35.125944 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.125853    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:35.137626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.137597    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 21:03:35.147931 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.147897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-config\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.147931 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.147941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-systemd\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-tuned\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-netns\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-ovn\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-modprobe-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.148178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-lib-modules\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-netns\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-cni-binary-copy\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-k8s-cni-cncf-io\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07af5c6-3d90-4231-9f57-63f3566f8f62-tmp-dir\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.148481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-k8s-cni-cncf-io\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-sys\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrdm\" (UniqueName: \"kubernetes.io/projected/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-kube-api-access-mbrdm\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-systemd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-script-lib\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-dbus\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ndl\" (UniqueName: \"kubernetes.io/projected/a6e6382c-dc81-4584-ac12-bd4a147c874f-kube-api-access-p7ndl\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de8b18af-2228-40dc-b006-01407c1c8ba7-konnectivity-ca\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-kubelet-config\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.148828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-run\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-socket-dir-parent\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-conf-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-daemon-config\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-serviceca\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-systemd-units\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-socket-dir-parent\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-netns\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.148990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-conf-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-socket-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-cni-binary-copy\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysconfig\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-etc-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hxw\" (UniqueName: \"kubernetes.io/projected/3f40bd58-fd66-415b-9dc8-a7235a1e1316-kube-api-access-z6hxw\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwf67\" (UniqueName: \"kubernetes.io/projected/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kube-api-access-gwf67\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.149310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9wtq\" (UniqueName: \"kubernetes.io/projected/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-kube-api-access-q9wtq\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-conf\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6e6382c-dc81-4584-ac12-bd4a147c874f-iptables-alerter-script\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6e6382c-dc81-4584-ac12-bd4a147c874f-host-slash\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-multus\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pz9\" (UniqueName: \"kubernetes.io/projected/23e94a11-f3fc-4b84-b714-3667c90e23c7-kube-api-access-l6pz9\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de8b18af-2228-40dc-b006-01407c1c8ba7-agent-certs\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gvh\" (UniqueName: \"kubernetes.io/projected/d380a421-1160-4abb-9afb-7bb0adff6d05-kube-api-access-24gvh\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6e6382c-dc81-4584-ac12-bd4a147c874f-host-slash\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de8b18af-2228-40dc-b006-01407c1c8ba7-konnectivity-ca\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-serviceca\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-env-overrides\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-multus\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-system-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-daemon-config\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-etc-kubernetes\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovn-node-metrics-cert\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-system-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-etc-kubernetes\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-registration-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149732    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-tmp\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgh6\" (UniqueName: \"kubernetes.io/projected/1f88745e-d889-4ba2-82ed-967013d9ba60-kube-api-access-5qgh6\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-system-cni-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-netd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-var-lib-kubelet\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a6e6382c-dc81-4584-ac12-bd4a147c874f-iptables-alerter-script\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.149996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-cnibin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-kubelet\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-cnibin\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.150919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-cnibin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-kubelet\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-kubelet\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-node-log\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-bin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-multus-cni-dir\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-var-lib-cni-bin\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-os-release\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-log-socket\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-var-lib-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-device-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-os-release\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-host\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-os-release\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-bin\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-host\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.151444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a07af5c6-3d90-4231-9f57-63f3566f8f62-hosts-file\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8dx\" (UniqueName: \"kubernetes.io/projected/a07af5c6-3d90-4231-9f57-63f3566f8f62-kube-api-access-qn8dx\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-kubernetes\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-host\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-sys-fs\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-hostroot\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-multus-certs\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-host-run-multus-certs\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23e94a11-f3fc-4b84-b714-3667c90e23c7-hostroot\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.152036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.150729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-slash\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.153564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.153541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de8b18af-2228-40dc-b006-01407c1c8ba7-agent-certs\") pod \"konnectivity-agent-8vmkk\" (UID: \"de8b18af-2228-40dc-b006-01407c1c8ba7\") " pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.154310 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.154290    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:35.154440 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.154316    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:35.154440 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.154330    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:35.154440 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.154423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:03:35.654399719 +0000 UTC m=+2.952877476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:35.157630 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.157602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrdm\" (UniqueName: \"kubernetes.io/projected/e2a2e0e9-8711-4116-a953-cfd4d2b5c978-kube-api-access-mbrdm\") pod \"node-ca-t9785\" (UID: \"e2a2e0e9-8711-4116-a953-cfd4d2b5c978\") " pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.157755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.157696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ndl\" (UniqueName: \"kubernetes.io/projected/a6e6382c-dc81-4584-ac12-bd4a147c874f-kube-api-access-p7ndl\") pod \"iptables-alerter-xhv5l\" (UID: \"a6e6382c-dc81-4584-ac12-bd4a147c874f\") " pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.158080 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.157613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pz9\" (UniqueName: \"kubernetes.io/projected/23e94a11-f3fc-4b84-b714-3667c90e23c7-kube-api-access-l6pz9\") pod \"multus-g4xnp\" (UID: \"23e94a11-f3fc-4b84-b714-3667c90e23c7\") " pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.186941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.186875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"2c807facfd39d2d720088d43e7e67d0a","Type":"ContainerStarted","Data":"de256c81d11fa0075099e5188f2e2236c4228281e1e3f43653db509a80193bca"}
Jul 29 21:03:35.188030 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.188003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" event={"ID":"df72579e9d2f6ec664c78ab551d5adbe","Type":"ContainerStarted","Data":"bf18f753c70fa422a5891923366051860583c1bbcf45de6d7dfc6cac98fa2325"}
Jul 29 21:03:35.252016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.251978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-kubernetes\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-host\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.252223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-host\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-kubernetes\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.252223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-sys-fs\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-sys-fs\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-slash\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-config\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-systemd\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-tuned\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-slash\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-ovn\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-systemd\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-modprobe-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-lib-modules\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07af5c6-3d90-4231-9f57-63f3566f8f62-tmp-dir\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-sys\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-ovn\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-systemd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.252659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-script-lib\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-dbus\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-kubelet-config\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-modprobe-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-run\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-systemd-units\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-netns\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-socket-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-lib-modules\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysconfig\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-etc-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07af5c6-3d90-4231-9f57-63f3566f8f62-tmp-dir\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.253386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hxw\" (UniqueName: \"kubernetes.io/projected/3f40bd58-fd66-415b-9dc8-a7235a1e1316-kube-api-access-z6hxw\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-etc-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwf67\" (UniqueName: \"kubernetes.io/projected/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kube-api-access-gwf67\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9wtq\" (UniqueName: \"kubernetes.io/projected/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-kube-api-access-q9wtq\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-conf\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gvh\" (UniqueName: \"kubernetes.io/projected/d380a421-1160-4abb-9afb-7bb0adff6d05-kube-api-access-24gvh\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-env-overrides\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-kubelet-config\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-config\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-systemd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.253354    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.253426    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:35.753411025 +0000 UTC m=+3.051888764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-socket-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-run\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.254259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-systemd-units\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysconfig\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-netns\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.252708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-sys\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.253726    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-dbus\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.253770    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:35.75375464 +0000 UTC m=+3.052232378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-conf\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovnkube-script-lib\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovn-node-metrics-cert\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-registration-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-tmp\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgh6\" (UniqueName: \"kubernetes.io/projected/1f88745e-d889-4ba2-82ed-967013d9ba60-kube-api-access-5qgh6\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.253992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-system-cni-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-netd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.255190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-var-lib-kubelet\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-cnibin\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-kubelet\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-node-log\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-os-release\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-log-socket\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-system-cni-dir\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-var-lib-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-device-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-bin\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a07af5c6-3d90-4231-9f57-63f3566f8f62-hosts-file\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8dx\" (UniqueName: \"kubernetes.io/projected/a07af5c6-3d90-4231-9f57-63f3566f8f62-kube-api-access-qn8dx\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-netd\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-sysctl-d\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-log-socket\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.256077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-var-lib-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f88745e-d889-4ba2-82ed-967013d9ba60-var-lib-kubelet\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-device-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-cnibin\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-node-log\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d380a421-1160-4abb-9afb-7bb0adff6d05-os-release\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-cni-bin\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c5a5cd9-e828-4de7-aef8-a6ef67688158-registration-dir\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.254901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-run-openvswitch\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.255030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f40bd58-fd66-415b-9dc8-a7235a1e1316-host-kubelet\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.255042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a07af5c6-3d90-4231-9f57-63f3566f8f62-hosts-file\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.255568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d380a421-1160-4abb-9afb-7bb0adff6d05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.255848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-etc-tuned\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.256278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f40bd58-fd66-415b-9dc8-a7235a1e1316-env-overrides\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.256949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f40bd58-fd66-415b-9dc8-a7235a1e1316-ovn-node-metrics-cert\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.257906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.257883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f88745e-d889-4ba2-82ed-967013d9ba60-tmp\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.264037 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.263540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9wtq\" (UniqueName: \"kubernetes.io/projected/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-kube-api-access-q9wtq\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.264037 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.263993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hxw\" (UniqueName: \"kubernetes.io/projected/3f40bd58-fd66-415b-9dc8-a7235a1e1316-kube-api-access-z6hxw\") pod \"ovnkube-node-2xwnk\" (UID: \"3f40bd58-fd66-415b-9dc8-a7235a1e1316\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.264237 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.264100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8dx\" (UniqueName: \"kubernetes.io/projected/a07af5c6-3d90-4231-9f57-63f3566f8f62-kube-api-access-qn8dx\") pod \"node-resolver-dn9wv\" (UID: \"a07af5c6-3d90-4231-9f57-63f3566f8f62\") " pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.264306 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.264284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gvh\" (UniqueName: \"kubernetes.io/projected/d380a421-1160-4abb-9afb-7bb0adff6d05-kube-api-access-24gvh\") pod \"multus-additional-cni-plugins-mk8nn\" (UID: \"d380a421-1160-4abb-9afb-7bb0adff6d05\") " pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.265249 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.265227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgh6\" (UniqueName: \"kubernetes.io/projected/1f88745e-d889-4ba2-82ed-967013d9ba60-kube-api-access-5qgh6\") pod \"tuned-z8hth\" (UID: \"1f88745e-d889-4ba2-82ed-967013d9ba60\") " pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.265455 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.265437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwf67\" (UniqueName: \"kubernetes.io/projected/2c5a5cd9-e828-4de7-aef8-a6ef67688158-kube-api-access-gwf67\") pod \"aws-ebs-csi-driver-node-6gj7f\" (UID: \"2c5a5cd9-e828-4de7-aef8-a6ef67688158\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.344856 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.344807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhv5l"
Jul 29 21:03:35.351949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.351916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4xnp"
Jul 29 21:03:35.362988 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.362956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:35.368974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.368946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t9785"
Jul 29 21:03:35.377007 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.376945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dn9wv"
Jul 29 21:03:35.385077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.385049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mk8nn"
Jul 29 21:03:35.392177 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.392148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:03:35.400286 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.400259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f"
Jul 29 21:03:35.406253 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.406223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z8hth"
Jul 29 21:03:35.584110 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.584085    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e94a11_f3fc_4b84_b714_3667c90e23c7.slice/crio-2bd658931301774f0ea7fd357357bec3d6e0e2bad69ea39195561e8acd9e27bb WatchSource:0}: Error finding container 2bd658931301774f0ea7fd357357bec3d6e0e2bad69ea39195561e8acd9e27bb: Status 404 returned error can't find the container with id 2bd658931301774f0ea7fd357357bec3d6e0e2bad69ea39195561e8acd9e27bb
Jul 29 21:03:35.584913 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.584891    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380a421_1160_4abb_9afb_7bb0adff6d05.slice/crio-e6e86d03d34bad8a4601c3f9aeb6d712d7d1e70f3957e865d9210bc0fa0be6a2 WatchSource:0}: Error finding container e6e86d03d34bad8a4601c3f9aeb6d712d7d1e70f3957e865d9210bc0fa0be6a2: Status 404 returned error can't find the container with id e6e86d03d34bad8a4601c3f9aeb6d712d7d1e70f3957e865d9210bc0fa0be6a2
Jul 29 21:03:35.585854 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.585828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8b18af_2228_40dc_b006_01407c1c8ba7.slice/crio-a72426b0c30f28cd8a4e774daf2c8c1b2f1bd5d6dd41bd6ad3c801f15a3ec935 WatchSource:0}: Error finding container a72426b0c30f28cd8a4e774daf2c8c1b2f1bd5d6dd41bd6ad3c801f15a3ec935: Status 404 returned error can't find the container with id a72426b0c30f28cd8a4e774daf2c8c1b2f1bd5d6dd41bd6ad3c801f15a3ec935
Jul 29 21:03:35.588384 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.587662    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f88745e_d889_4ba2_82ed_967013d9ba60.slice/crio-799e036a0a7156bbd053f47476f242c47635c14694cf282ce1dba7360cc2f405 WatchSource:0}: Error finding container 799e036a0a7156bbd053f47476f242c47635c14694cf282ce1dba7360cc2f405: Status 404 returned error can't find the container with id 799e036a0a7156bbd053f47476f242c47635c14694cf282ce1dba7360cc2f405
Jul 29 21:03:35.591710 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.591266    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5a5cd9_e828_4de7_aef8_a6ef67688158.slice/crio-7059bd1bb7251218317f9b8ee1ca83dc86258285385e1707dae1ebcfd1cb8dab WatchSource:0}: Error finding container 7059bd1bb7251218317f9b8ee1ca83dc86258285385e1707dae1ebcfd1cb8dab: Status 404 returned error can't find the container with id 7059bd1bb7251218317f9b8ee1ca83dc86258285385e1707dae1ebcfd1cb8dab
Jul 29 21:03:35.592157 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.591830    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a2e0e9_8711_4116_a953_cfd4d2b5c978.slice/crio-94bf02f18b6f0aff3bb69bf352ac4cab4686a6ac76c6571c91bc1b8200cd0ab5 WatchSource:0}: Error finding container 94bf02f18b6f0aff3bb69bf352ac4cab4686a6ac76c6571c91bc1b8200cd0ab5: Status 404 returned error can't find the container with id 94bf02f18b6f0aff3bb69bf352ac4cab4686a6ac76c6571c91bc1b8200cd0ab5
Jul 29 21:03:35.592317 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.592280    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e6382c_dc81_4584_ac12_bd4a147c874f.slice/crio-5c6064814b28d82ccfff38e348b1b2fad61b7f4915a2f64a2bb58fc619ca1854 WatchSource:0}: Error finding container 5c6064814b28d82ccfff38e348b1b2fad61b7f4915a2f64a2bb58fc619ca1854: Status 404 returned error can't find the container with id 5c6064814b28d82ccfff38e348b1b2fad61b7f4915a2f64a2bb58fc619ca1854
Jul 29 21:03:35.593055 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:03:35.593028    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f40bd58_fd66_415b_9dc8_a7235a1e1316.slice/crio-b03bc8efec52ed2255f7c04f8aeb89664ac34354d170775f5277e019525b2868 WatchSource:0}: Error finding container b03bc8efec52ed2255f7c04f8aeb89664ac34354d170775f5277e019525b2868: Status 404 returned error can't find the container with id b03bc8efec52ed2255f7c04f8aeb89664ac34354d170775f5277e019525b2868
Jul 29 21:03:35.657376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.657309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:35.657557 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.657499    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:35.657557 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.657524    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:35.657557 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.657537    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:35.657698 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.657601    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:03:36.657585101 +0000 UTC m=+3.956062836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:35.758367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.758327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:35.758367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:35.758371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:35.758667 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.758496    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:35.758667 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.758507    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:35.758667 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.758559    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:36.758545288 +0000 UTC m=+4.057023023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:35.758667 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:35.758579    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:36.758568513 +0000 UTC m=+4.057046248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:36.085793 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.085449    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:58:35 +0000 UTC" deadline="2028-03-07 23:54:57.236910853 +0000 UTC"
Jul 29 21:03:36.085793 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.085788    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14090h51m21.151132139s"
Jul 29 21:03:36.195012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.193963    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c807facfd39d2d720088d43e7e67d0a" containerID="68396e58499f152e33e4c65e13c2eabbaee6f66fa42914265765f776053bcc1d" exitCode=0
Jul 29 21:03:36.195012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.194059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"2c807facfd39d2d720088d43e7e67d0a","Type":"ContainerDied","Data":"68396e58499f152e33e4c65e13c2eabbaee6f66fa42914265765f776053bcc1d"}
Jul 29 21:03:36.214743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.214040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"b03bc8efec52ed2255f7c04f8aeb89664ac34354d170775f5277e019525b2868"}
Jul 29 21:03:36.215743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.215715    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:03:36.226133 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.226098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z8hth" event={"ID":"1f88745e-d889-4ba2-82ed-967013d9ba60","Type":"ContainerStarted","Data":"799e036a0a7156bbd053f47476f242c47635c14694cf282ce1dba7360cc2f405"}
Jul 29 21:03:36.229538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.229425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerStarted","Data":"e6e86d03d34bad8a4601c3f9aeb6d712d7d1e70f3957e865d9210bc0fa0be6a2"}
Jul 29 21:03:36.235221 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.235182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4xnp" event={"ID":"23e94a11-f3fc-4b84-b714-3667c90e23c7","Type":"ContainerStarted","Data":"2bd658931301774f0ea7fd357357bec3d6e0e2bad69ea39195561e8acd9e27bb"}
Jul 29 21:03:36.243164 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.243080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhv5l" event={"ID":"a6e6382c-dc81-4584-ac12-bd4a147c874f","Type":"ContainerStarted","Data":"5c6064814b28d82ccfff38e348b1b2fad61b7f4915a2f64a2bb58fc619ca1854"}
Jul 29 21:03:36.261630 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.256361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t9785" event={"ID":"e2a2e0e9-8711-4116-a953-cfd4d2b5c978","Type":"ContainerStarted","Data":"94bf02f18b6f0aff3bb69bf352ac4cab4686a6ac76c6571c91bc1b8200cd0ab5"}
Jul 29 21:03:36.282579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.282256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f" event={"ID":"2c5a5cd9-e828-4de7-aef8-a6ef67688158","Type":"ContainerStarted","Data":"7059bd1bb7251218317f9b8ee1ca83dc86258285385e1707dae1ebcfd1cb8dab"}
Jul 29 21:03:36.297212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.297091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vmkk" event={"ID":"de8b18af-2228-40dc-b006-01407c1c8ba7","Type":"ContainerStarted","Data":"a72426b0c30f28cd8a4e774daf2c8c1b2f1bd5d6dd41bd6ad3c801f15a3ec935"}
Jul 29 21:03:36.300882 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.300847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dn9wv" event={"ID":"a07af5c6-3d90-4231-9f57-63f3566f8f62","Type":"ContainerStarted","Data":"26d5920445a510ac6f678b5d24ba6e701645760ee9a383d8f1f5d98c3e5cef9b"}
Jul 29 21:03:36.664491 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.664388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:36.664665 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.664607    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:36.664665 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.664629    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:36.664665 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.664643    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:36.664877 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.664767    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:03:38.664725525 +0000 UTC m=+5.963203278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.765644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:36.765699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.765933    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.766003    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:38.765983478 +0000 UTC m=+6.064461217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.766474    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:36.766575 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:36.766532    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:38.76651409 +0000 UTC m=+6.064991828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:37.086405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.086269    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:58:35 +0000 UTC" deadline="2028-04-24 10:07:56.51811473 +0000 UTC"
Jul 29 21:03:37.086405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.086345    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15229h4m19.431774307s"
Jul 29 21:03:37.184106 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.184066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:37.184310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.184111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:37.184310 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:37.184237    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:37.184488 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.184469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:37.184659 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:37.184597    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:37.184835 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:37.184703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:37.319239 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:37.319197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" event={"ID":"2c807facfd39d2d720088d43e7e67d0a","Type":"ContainerStarted","Data":"4880dea70b2b8154912bdc8e74f6110da8857a80a6eacddcfe7765aa9f516066"}
Jul 29 21:03:38.684025 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.683989    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:38.684025 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.684030    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:38.684567 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.684046    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:38.684567 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.684119    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:03:42.684095255 +0000 UTC m=+9.982572995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:38.684768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:38.683708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:38.786006 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:38.785967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:38.786180 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:38.786023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:38.786265 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.786192    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:38.786305 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.786268    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:42.786248587 +0000 UTC m=+10.084726322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:38.786751 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.786730    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:38.786848 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:38.786814    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:42.786795748 +0000 UTC m=+10.085273498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:39.181925 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:39.181835    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:39.181925 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:39.181870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:39.182153 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:39.181977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:39.182407 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:39.182380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:39.182513 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:39.182434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:39.182567 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:39.182513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:41.181597 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:41.181553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:41.182091 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:41.181703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:41.182204 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:41.182185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:41.182337 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:41.182313    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:41.184838 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:41.182422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:41.190500 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:41.189498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:42.723273 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:42.723230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:42.723784 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.723437    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:42.723784 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.723463    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:42.723784 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.723479    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:42.723784 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.723550    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:03:50.723527065 +0000 UTC m=+18.022004817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:42.824010 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:42.823945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:42.824010 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:42.824004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:42.824264 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.824145    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:42.824264 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.824217    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:50.824197218 +0000 UTC m=+18.122674978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:42.824618 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.824598    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:42.824703 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:42.824651    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:03:50.824638508 +0000 UTC m=+18.123116243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:43.181743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:43.181630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:43.181743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:43.181732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:43.181990 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:43.181748    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:43.181990 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:43.181854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:43.181990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:43.181905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:43.181990 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:43.181979    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:45.183835    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:45.183841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:45.183985    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:45.183853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:45.184067    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:45.184336 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:45.184121    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:47.181558 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:47.181469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:47.182026 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:47.181472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:47.182026 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:47.181612    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:47.182026 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:47.181488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:47.182026 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:47.181687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:47.182026 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:47.181756    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:49.180969 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:49.180934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:49.180969 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:49.180934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:49.181485 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:49.181071    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:49.181485 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:49.181126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:49.181485 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:49.181229    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:49.181485 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:49.181324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:50.785715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:50.785664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:50.786141 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.785875    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:03:50.786141 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.785906    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:03:50.786141 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.785920    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:50.786141 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.785989    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:04:06.785967897 +0000 UTC m=+34.084445650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:03:50.886444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:50.886399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:50.886444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:50.886448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:50.886676 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.886563    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:50.886676 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.886563    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:50.886676 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.886631    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:06.886615046 +0000 UTC m=+34.185092779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:03:50.886676 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:50.886646    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:06.88663992 +0000 UTC m=+34.185117652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:03:51.181499 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:51.181464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:51.181687 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:51.181602    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:51.181687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:51.181464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:51.181857 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:51.181713    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:51.181857 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:51.181465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:51.181950 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:51.181870    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:53.182344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:53.182305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:53.182850 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:53.182444    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:53.182850 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:53.182511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:53.182850 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:53.182581    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:53.183029 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:53.182953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:53.183079 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:53.183033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:55.181670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:55.181671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:55.181847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:55.181837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:55.181981    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:55.182213 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:55.182061    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:56.370735 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.370297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f" event={"ID":"2c5a5cd9-e828-4de7-aef8-a6ef67688158","Type":"ContainerStarted","Data":"622c54380de25be3eb17cdcfaec633393001581f900c072369ff8b1cb311372d"}
Jul 29 21:03:56.371951 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.371910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vmkk" event={"ID":"de8b18af-2228-40dc-b006-01407c1c8ba7","Type":"ContainerStarted","Data":"6d3a81501a806f24843734766e05664292921e85898614c53e83ae98b480a66a"}
Jul 29 21:03:56.373645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.373623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dn9wv" event={"ID":"a07af5c6-3d90-4231-9f57-63f3566f8f62","Type":"ContainerStarted","Data":"bf4c7aa31d211d9c91a0d7c509413d7af9d0dd1f21c6d7e91df251b86b907b0b"}
Jul 29 21:03:56.375214 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.375189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" event={"ID":"df72579e9d2f6ec664c78ab551d5adbe","Type":"ContainerStarted","Data":"7efe435e691bc67bcad22eae10f6fb8a581208364098131bf111fadfc7fa6a9d"}
Jul 29 21:03:56.378168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378146    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:03:56.378523 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378501    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f40bd58-fd66-415b-9dc8-a7235a1e1316" containerID="425356f331c43f1c1c888e01a71b8afa0bf4c0d5142a680a712ef4af1bfa33e1" exitCode=1
Jul 29 21:03:56.378640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"759b14e0e1d82b7003b5b88e842aa1e6be4ca58939cfea4db29ab10265445a23"}
Jul 29 21:03:56.378640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"ad05fc19a4240f41ab590203be3140ba9065a0dfd7142dec29014f8ea24ef366"}
Jul 29 21:03:56.378783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"6ba241af0c152a982f1e962deeffa8e8e7814fbe6901cc8574e7465b19f864a0"}
Jul 29 21:03:56.378783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"c4c5850487f98e97c34ebc300ba306dfb828b336fa079ebf8db0bc1f53988afb"}
Jul 29 21:03:56.378783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerDied","Data":"425356f331c43f1c1c888e01a71b8afa0bf4c0d5142a680a712ef4af1bfa33e1"}
Jul 29 21:03:56.378934 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.378813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"025e1081bf909b00b34fa541cd9480186fe33ef7fcca1be3c86251ed153b1af1"}
Jul 29 21:03:56.380182 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.380144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z8hth" event={"ID":"1f88745e-d889-4ba2-82ed-967013d9ba60","Type":"ContainerStarted","Data":"ee9cf91284d36ba0c5fde08fa9d45aafdd8fc7c7cad5ad65d21a9faff0fb3db0"}
Jul 29 21:03:56.381788 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.381768    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="cfda75ef6a70a32aba36976cc2d562169db63a66b7fc26b6f7b2c858bca6dbe0" exitCode=0
Jul 29 21:03:56.381889 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.381852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"cfda75ef6a70a32aba36976cc2d562169db63a66b7fc26b6f7b2c858bca6dbe0"}
Jul 29 21:03:56.384384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.384358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4xnp" event={"ID":"23e94a11-f3fc-4b84-b714-3667c90e23c7","Type":"ContainerStarted","Data":"d527673d61d2d10f8da95e56c5ba52283b4e04c7722a2a997a3eec8e782d742f"}
Jul 29 21:03:56.386096 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.386075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t9785" event={"ID":"e2a2e0e9-8711-4116-a953-cfd4d2b5c978","Type":"ContainerStarted","Data":"7cf2bad79f840f1cb7af273069cc01800d476f3c87587236684b9545347f17d3"}
Jul 29 21:03:56.387968 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.387928    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-168.ec2.internal" podStartSLOduration=21.387916477 podStartE2EDuration="21.387916477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:37.33887373 +0000 UTC m=+4.637351491" watchObservedRunningTime="2026-07-29 21:03:56.387916477 +0000 UTC m=+23.686394232"
Jul 29 21:03:56.404319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.404267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8vmkk" podStartSLOduration=2.45209273 podStartE2EDuration="22.404248749s" totalImagesPullingTime="19.952156019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.388346952 +0000 UTC m=+23.686824707" watchObservedRunningTime="2026-07-29 21:03:56.404248749 +0000 UTC m=+23.702726504"
Jul 29 21:03:56.404491 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.404357    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t9785" podStartSLOduration=2.730822376 podStartE2EDuration="22.40435168s" totalImagesPullingTime="19.673529304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.403767472 +0000 UTC m=+23.702245228" watchObservedRunningTime="2026-07-29 21:03:56.40435168 +0000 UTC m=+23.702829434"
Jul 29 21:03:56.460132 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.460059    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dn9wv" podStartSLOduration=2.751973648 podStartE2EDuration="22.460035268s" totalImagesPullingTime="19.70806162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.459797783 +0000 UTC m=+23.758275539" watchObservedRunningTime="2026-07-29 21:03:56.460035268 +0000 UTC m=+23.758513026"
Jul 29 21:03:56.479079 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.479016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z8hth" podStartSLOduration=2.735612077 podStartE2EDuration="22.47899385s" totalImagesPullingTime="19.743381773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.478788127 +0000 UTC m=+23.777265883" watchObservedRunningTime="2026-07-29 21:03:56.47899385 +0000 UTC m=+23.777471606"
Jul 29 21:03:56.498022 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.497960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g4xnp" podStartSLOduration=2.632381956 podStartE2EDuration="22.497939262s" totalImagesPullingTime="19.865557306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.497888682 +0000 UTC m=+23.796366437" watchObservedRunningTime="2026-07-29 21:03:56.497939262 +0000 UTC m=+23.796417018"
Jul 29 21:03:56.534541 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.534489    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-168.ec2.internal" podStartSLOduration=21.53447238 podStartE2EDuration="21.53447238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:56.534332077 +0000 UTC m=+23.832809842" watchObservedRunningTime="2026-07-29 21:03:56.53447238 +0000 UTC m=+23.832950135"
Jul 29 21:03:56.990469 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:56.990435    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 21:03:57.120815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.120678    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T21:03:56.990461916Z","UUID":"42928fb1-cbe3-4fa6-a52b-c34eaed673c8","Handler":null,"Name":"","Endpoint":""}
Jul 29 21:03:57.122430 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.122407    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 21:03:57.122568 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.122461    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 21:03:57.181577 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.181543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:57.181577 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.181573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:57.181822 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.181543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:57.181822 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:57.181673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:57.181822 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:57.181788    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:57.181942 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:57.181876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:57.390956 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.390854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhv5l" event={"ID":"a6e6382c-dc81-4584-ac12-bd4a147c874f","Type":"ContainerStarted","Data":"566c2245006a81109303a4eb42ed050ee04f181b121e2e497b6d77046e7eed87"}
Jul 29 21:03:57.393268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.393221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f" event={"ID":"2c5a5cd9-e828-4de7-aef8-a6ef67688158","Type":"ContainerStarted","Data":"8d12435b047c4b070c86cf47f6d0f4e8915f80466543c8f6547d7773156c9dc7"}
Jul 29 21:03:57.406691 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:57.406646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xhv5l" podStartSLOduration=4.5734335040000005 podStartE2EDuration="24.406631348s" totalImagesPullingTime="19.833197844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:57.406111282 +0000 UTC m=+24.704589038" watchObservedRunningTime="2026-07-29 21:03:57.406631348 +0000 UTC m=+24.705109103"
Jul 29 21:03:58.400248 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:58.400221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:03:58.400717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:58.400664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"9d971387f14261bc3a3c62a018cffe9c19bc3d422ffaf3545038b4c746c16c1a"}
Jul 29 21:03:58.403551 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:58.403484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f" event={"ID":"2c5a5cd9-e828-4de7-aef8-a6ef67688158","Type":"ContainerStarted","Data":"a46e27e65e539f1bcdef2f3bdc7e47bbc68eb69795f64f32414cef0be9c37935"}
Jul 29 21:03:58.421198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:58.421147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gj7f" podStartSLOduration=2.512164419 podStartE2EDuration="24.421126446s" totalImagesPullingTime="21.908962027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:03:58.420176439 +0000 UTC m=+25.718654203" watchObservedRunningTime="2026-07-29 21:03:58.421126446 +0000 UTC m=+25.719604202"
Jul 29 21:03:59.181166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:59.180939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:03:59.181347 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:59.180967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:03:59.181347 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:59.181268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:03:59.181347 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:59.181339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:03:59.181513 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:59.180967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:03:59.181513 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:03:59.181416    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:03:59.980081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:59.979463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:03:59.980798 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:03:59.980777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:04:00.409514 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:00.409485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:04:00.410092 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:00.410071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vmkk"
Jul 29 21:04:01.181492 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.181271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:01.182326 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.181274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:01.182326 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:01.181564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:04:01.182326 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:01.181632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:04:01.182326 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.181334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:01.182326 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:01.181728    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:04:01.415487 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.415459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:04:01.415862 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.415833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"af2d12212ea1ee710d712b3461dfcb3a2b4aec7f7454449388a7f6f102c5878e"}
Jul 29 21:04:01.416229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.416134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:01.416229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.416198    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:01.416461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.416328    2581 scope.go:117] "RemoveContainer" containerID="425356f331c43f1c1c888e01a71b8afa0bf4c0d5142a680a712ef4af1bfa33e1"
Jul 29 21:04:01.417990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.417962    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="05033cbf1e3a895db08ff35cfd24e609204614f86f72cfdc26a92e9eb9492506" exitCode=0
Jul 29 21:04:01.418086 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.417992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"05033cbf1e3a895db08ff35cfd24e609204614f86f72cfdc26a92e9eb9492506"}
Jul 29 21:04:01.434974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:01.434949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:02.425181 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.425093    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:04:02.425672 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.425509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" event={"ID":"3f40bd58-fd66-415b-9dc8-a7235a1e1316","Type":"ContainerStarted","Data":"3ec7bedf900f933382d38045452151a7c4eb1fac7ebd07f0e056734dd5b46d61"}
Jul 29 21:04:02.425769 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.425740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:02.428083 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.428054    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="0770a9664c1e522f66aae235afb695bedc6771e81d0f4f57a9e595dfaa6e7f74" exitCode=0
Jul 29 21:04:02.428216 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.428135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"0770a9664c1e522f66aae235afb695bedc6771e81d0f4f57a9e595dfaa6e7f74"}
Jul 29 21:04:02.443146 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.443119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:02.459393 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.459340    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk" podStartSLOduration=8.626325121 podStartE2EDuration="28.45932056s" totalImagesPullingTime="19.832995439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:04:02.459074675 +0000 UTC m=+29.757552429" watchObservedRunningTime="2026-07-29 21:04:02.45932056 +0000 UTC m=+29.757798363"
Jul 29 21:04:02.961089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.961041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tv28"]
Jul 29 21:04:02.961254 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.961220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:02.961371 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:02.961347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:04:02.961745 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.961720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4q5p"]
Jul 29 21:04:02.961875 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.961820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:02.962407 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:02.961899    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:04:02.963890 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.963860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rtcfx"]
Jul 29 21:04:02.964043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:02.964025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:02.964182 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:02.964154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:04:03.142536 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:04:03.142359    2581 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380a421_1160_4abb_9afb_7bb0adff6d05.slice/crio-5c08c1405cf3fad6067dc402d549b237ed40153d9be89d06d689df45d9d7a665.scope/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380a421_1160_4abb_9afb_7bb0adff6d05.slice/crio-5c08c1405cf3fad6067dc402d549b237ed40153d9be89d06d689df45d9d7a665.scope/cpuset.cpus.effective: no such device
Jul 29 21:04:03.150122 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:03.150091    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380a421_1160_4abb_9afb_7bb0adff6d05.slice/crio-5c08c1405cf3fad6067dc402d549b237ed40153d9be89d06d689df45d9d7a665.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 21:04:03.432410 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:03.432379    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="5c08c1405cf3fad6067dc402d549b237ed40153d9be89d06d689df45d9d7a665" exitCode=0
Jul 29 21:04:03.432847 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:03.432462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"5c08c1405cf3fad6067dc402d549b237ed40153d9be89d06d689df45d9d7a665"}
Jul 29 21:04:05.181195 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:05.181110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:05.181876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:05.181118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:05.181876 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:05.181258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:04:05.181876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:05.181117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:05.181876 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:05.181340    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:04:05.181876 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:05.181396    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:04:06.805804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:06.805744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:06.806296 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.805944    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:04:06.806296 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.805974    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:04:06.806296 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.805991    2581 projected.go:196] Error preparing data for projected volume kube-api-access-r9knh for pod openshift-network-diagnostics/network-check-target-rtcfx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:04:06.806296 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.806064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh podName:4dfae41e-2056-4ecc-916c-a9d2f14b8f2a nodeName:}" failed. No retries permitted until 2026-07-29 21:04:38.806046414 +0000 UTC m=+66.104524148 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9knh" (UniqueName: "kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh") pod "network-check-target-rtcfx" (UID: "4dfae41e-2056-4ecc-916c-a9d2f14b8f2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:04:06.907166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:06.907129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:06.907336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:06.907178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:06.907336 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.907268    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:04:06.907336 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.907295    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:04:06.907474 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.907351    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:38.907328867 +0000 UTC m=+66.205806601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:04:06.907474 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:06.907371    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret podName:deb2dcc5-86f5-4cb7-be2a-826aa8b04b60 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:38.907362025 +0000 UTC m=+66.205839761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret") pod "global-pull-secret-syncer-t4q5p" (UID: "deb2dcc5-86f5-4cb7-be2a-826aa8b04b60") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:04:07.181224 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:07.181188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:07.181413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:07.181194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:07.181413 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:07.181338    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:04:07.181413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:07.181193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:07.181529 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:07.181419    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t4q5p" podUID="deb2dcc5-86f5-4cb7-be2a-826aa8b04b60"
Jul 29 21:04:07.181529 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:07.181480    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtcfx" podUID="4dfae41e-2056-4ecc-916c-a9d2f14b8f2a"
Jul 29 21:04:08.446412 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.446379    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="f2f12a3744dfc873ad1f6e54e96a7cc9c7024bf9ae1a17f27f06c1d2510ee245" exitCode=0
Jul 29 21:04:08.446412 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.446418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"f2f12a3744dfc873ad1f6e54e96a7cc9c7024bf9ae1a17f27f06c1d2510ee245"}
Jul 29 21:04:08.587323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.587296    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-168.ec2.internal" event="NodeReady"
Jul 29 21:04:08.587471 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.587458    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 21:04:08.638731 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.638296    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8hsh"]
Jul 29 21:04:08.657984 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.657954    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jzv4d"]
Jul 29 21:04:08.658125 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.658072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.660611 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.660590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5hxhk\""
Jul 29 21:04:08.660767 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.660587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 21:04:08.660767 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.660594    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 21:04:08.678383 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.678344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jzv4d"]
Jul 29 21:04:08.678383 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.678381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8hsh"]
Jul 29 21:04:08.678554 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.678462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.680858 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.680831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 21:04:08.681016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.680839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 21:04:08.681016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.680941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 21:04:08.681288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.681275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tnkqz\""
Jul 29 21:04:08.721789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.721761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.721908 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.721794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzr2q\" (UniqueName: \"kubernetes.io/projected/72d5e612-cb4c-4305-bfbf-931fa65818c7-kube-api-access-gzr2q\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.721908 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.721814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d5e612-cb4c-4305-bfbf-931fa65818c7-config-volume\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.721908 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.721835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5e612-cb4c-4305-bfbf-931fa65818c7-tmp-dir\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzr2q\" (UniqueName: \"kubernetes.io/projected/72d5e612-cb4c-4305-bfbf-931fa65818c7-kube-api-access-gzr2q\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823290 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:08.823196    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:08.823290 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d5e612-cb4c-4305-bfbf-931fa65818c7-config-volume\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823290 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cd7f0-fea6-4130-a158-a141dfff3207-cert\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.823290 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:08.823261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:09.323238119 +0000 UTC m=+36.621715856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:08.823475 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89cj\" (UniqueName: \"kubernetes.io/projected/b32cd7f0-fea6-4130-a158-a141dfff3207-kube-api-access-j89cj\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.823475 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5e612-cb4c-4305-bfbf-931fa65818c7-tmp-dir\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5e612-cb4c-4305-bfbf-931fa65818c7-tmp-dir\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.823769 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.823754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d5e612-cb4c-4305-bfbf-931fa65818c7-config-volume\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.833176 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.833151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzr2q\" (UniqueName: \"kubernetes.io/projected/72d5e612-cb4c-4305-bfbf-931fa65818c7-kube-api-access-gzr2q\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:08.924208 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.924170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cd7f0-fea6-4130-a158-a141dfff3207-cert\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.924208 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.924209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89cj\" (UniqueName: \"kubernetes.io/projected/b32cd7f0-fea6-4130-a158-a141dfff3207-kube-api-access-j89cj\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.926849 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.926828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cd7f0-fea6-4130-a158-a141dfff3207-cert\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:08.932228 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:08.932204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89cj\" (UniqueName: \"kubernetes.io/projected/b32cd7f0-fea6-4130-a158-a141dfff3207-kube-api-access-j89cj\") pod \"ingress-canary-jzv4d\" (UID: \"b32cd7f0-fea6-4130-a158-a141dfff3207\") " pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:09.011538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.011450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jzv4d"
Jul 29 21:04:09.184634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.184445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:09.184830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.184496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:09.184830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.184496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:09.187532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.187506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 21:04:09.187532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.187521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 21:04:09.187789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.187580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cnnw5\""
Jul 29 21:04:09.187789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.187526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 21:04:09.188406 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.188388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 21:04:09.188487 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.188388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82zmz\""
Jul 29 21:04:09.214498 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.214473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jzv4d"]
Jul 29 21:04:09.218284 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:04:09.218251    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32cd7f0_fea6_4130_a158_a141dfff3207.slice/crio-c16c72ac3b090c9c79f4eeb079c8681be33384458db28b624baece82ad226bf2 WatchSource:0}: Error finding container c16c72ac3b090c9c79f4eeb079c8681be33384458db28b624baece82ad226bf2: Status 404 returned error can't find the container with id c16c72ac3b090c9c79f4eeb079c8681be33384458db28b624baece82ad226bf2
Jul 29 21:04:09.327832 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.327724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:09.327977 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:09.327848    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:09.327977 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:09.327916    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:10.327897964 +0000 UTC m=+37.626375699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:09.451775 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.451745    2581 generic.go:356] "Generic (PLEG): container finished" podID="d380a421-1160-4abb-9afb-7bb0adff6d05" containerID="46cb23feb7628e7f8eb15db0524869cc68c3c6d60afb1c60461d6b3139034291" exitCode=0
Jul 29 21:04:09.452263 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.451829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerDied","Data":"46cb23feb7628e7f8eb15db0524869cc68c3c6d60afb1c60461d6b3139034291"}
Jul 29 21:04:09.453009 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:09.452987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jzv4d" event={"ID":"b32cd7f0-fea6-4130-a158-a141dfff3207","Type":"ContainerStarted","Data":"c16c72ac3b090c9c79f4eeb079c8681be33384458db28b624baece82ad226bf2"}
Jul 29 21:04:10.336331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:10.336300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:10.336530 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:10.336477    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:10.336600 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:10.336560    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:12.336539394 +0000 UTC m=+39.635017134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:10.463217 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:10.463178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" event={"ID":"d380a421-1160-4abb-9afb-7bb0adff6d05","Type":"ContainerStarted","Data":"b5cfcf9ed798681d24ffcb0b0d94dd38d4175bd2e91333737a3845f4f9585d80"}
Jul 29 21:04:10.485625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:10.485567    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mk8nn" podStartSLOduration=6.796415762 podStartE2EDuration="36.485550392s" totalImagesPullingTime="29.68913463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:04:10.483981307 +0000 UTC m=+37.782459103" watchObservedRunningTime="2026-07-29 21:04:10.485550392 +0000 UTC m=+37.784028148"
Jul 29 21:04:12.352200 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:12.352036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:12.352499 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:12.352214    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:12.352499 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:12.352279    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:16.35226307 +0000 UTC m=+43.650740803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:12.469929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:12.469897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jzv4d" event={"ID":"b32cd7f0-fea6-4130-a158-a141dfff3207","Type":"ContainerStarted","Data":"d08f742b76c3fc65f227acc362adc109a84d5b092cd3445ce28ba5a2152d0d7d"}
Jul 29 21:04:12.487870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:12.487820    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jzv4d" podStartSLOduration=1.7588121939999999 podStartE2EDuration="4.487798166s" totalImagesPullingTime="2.728985972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:04:12.486510265 +0000 UTC m=+39.784988020" watchObservedRunningTime="2026-07-29 21:04:12.487798166 +0000 UTC m=+39.786275919"
Jul 29 21:04:16.381974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:16.381927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:16.382500 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:16.382059    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:16.382500 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:16.382115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:24.382100761 +0000 UTC m=+51.680578494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:24.443891 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:24.443855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:24.444344 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:24.444008    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:24.444344 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:24.444074    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:04:40.444055813 +0000 UTC m=+67.742533545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:34.446579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:34.446550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xwnk"
Jul 29 21:04:38.847986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.847942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:38.851016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.850998    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 21:04:38.861449 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.861430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 21:04:38.873276 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.873245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9knh\" (UniqueName: \"kubernetes.io/projected/4dfae41e-2056-4ecc-916c-a9d2f14b8f2a-kube-api-access-r9knh\") pod \"network-check-target-rtcfx\" (UID: \"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a\") " pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:38.904729 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.904682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cnnw5\""
Jul 29 21:04:38.912636 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.912616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:38.948922 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.948845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:04:38.948922 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.948880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:38.951435 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.951409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 21:04:38.951531 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.951433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 21:04:38.959280 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:38.959254    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 21:04:38.959390 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:38.959338    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:05:42.959313511 +0000 UTC m=+130.257791259 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : secret "metrics-daemon-secret" not found
Jul 29 21:04:38.961680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:38.961656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/deb2dcc5-86f5-4cb7-be2a-826aa8b04b60-original-pull-secret\") pod \"global-pull-secret-syncer-t4q5p\" (UID: \"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60\") " pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:39.085477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:39.085440    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rtcfx"]
Jul 29 21:04:39.088752 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:04:39.088682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfae41e_2056_4ecc_916c_a9d2f14b8f2a.slice/crio-4d6f56b74f329ab52a8885fae2d1d5ee63aab69c74f338f2fd3bf558f8a1a5cd WatchSource:0}: Error finding container 4d6f56b74f329ab52a8885fae2d1d5ee63aab69c74f338f2fd3bf558f8a1a5cd: Status 404 returned error can't find the container with id 4d6f56b74f329ab52a8885fae2d1d5ee63aab69c74f338f2fd3bf558f8a1a5cd
Jul 29 21:04:39.207606 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:39.207521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t4q5p"
Jul 29 21:04:39.359919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:39.359888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t4q5p"]
Jul 29 21:04:39.363000 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:04:39.362974    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb2dcc5_86f5_4cb7_be2a_826aa8b04b60.slice/crio-cc6200d3d4bb1cbaa91d5cb89f88fe33c277d087c2e9804e2e8b2bca5a87c892 WatchSource:0}: Error finding container cc6200d3d4bb1cbaa91d5cb89f88fe33c277d087c2e9804e2e8b2bca5a87c892: Status 404 returned error can't find the container with id cc6200d3d4bb1cbaa91d5cb89f88fe33c277d087c2e9804e2e8b2bca5a87c892
Jul 29 21:04:39.537593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:39.537513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4q5p" event={"ID":"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60","Type":"ContainerStarted","Data":"cc6200d3d4bb1cbaa91d5cb89f88fe33c277d087c2e9804e2e8b2bca5a87c892"}
Jul 29 21:04:39.538504 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:39.538481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rtcfx" event={"ID":"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a","Type":"ContainerStarted","Data":"4d6f56b74f329ab52a8885fae2d1d5ee63aab69c74f338f2fd3bf558f8a1a5cd"}
Jul 29 21:04:40.459806 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:40.459766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:04:40.460390 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:40.459892    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:04:40.460390 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:04:40.459954    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:05:12.459935925 +0000 UTC m=+99.758413663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:04:42.548900 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:42.548857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rtcfx" event={"ID":"4dfae41e-2056-4ecc-916c-a9d2f14b8f2a","Type":"ContainerStarted","Data":"69e9bab72f0103860bb82d36b4ab5e5e91e2033263da036d7b4abc51a638cf9d"}
Jul 29 21:04:42.549334 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:42.548992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:04:42.564830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:42.564772    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rtcfx" podStartSLOduration=66.82624656 podStartE2EDuration="1m9.564749785s" totalImagesPullingTime="2.738503225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:04:42.56350624 +0000 UTC m=+69.861984008" watchObservedRunningTime="2026-07-29 21:04:42.564749785 +0000 UTC m=+69.863227543"
Jul 29 21:04:45.559098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:45.559052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t4q5p" event={"ID":"deb2dcc5-86f5-4cb7-be2a-826aa8b04b60","Type":"ContainerStarted","Data":"99c2336486a949945245f809815b6a5b0f23835f7a11b679a5d58cfaab27883a"}
Jul 29 21:04:45.575382 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:04:45.575322    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t4q5p" podStartSLOduration=65.610151063 podStartE2EDuration="1m11.575306135s" totalImagesPullingTime="5.965155072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:04:45.573785331 +0000 UTC m=+72.872263099" watchObservedRunningTime="2026-07-29 21:04:45.575306135 +0000 UTC m=+72.873783889"
Jul 29 21:05:12.479271 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:05:12.479222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:05:12.479681 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:05:12.479345    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:05:12.479681 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:05:12.479403    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:06:16.479388359 +0000 UTC m=+163.777866092 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:05:13.554175 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:05:13.554142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rtcfx"
Jul 29 21:05:42.993611 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:05:42.993563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:05:42.994132 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:05:42.993757    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 21:05:42.994132 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:05:42.993836    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:07:44.99381944 +0000 UTC m=+252.292297173 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : secret "metrics-daemon-secret" not found
Jul 29 21:06:08.281523 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.281490    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"]
Jul 29 21:06:08.283134 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.283118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.284834 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.284810    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"]
Jul 29 21:06:08.285738 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.285714    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 21:06:08.285863 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.285846    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 21:06:08.286486 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.286473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.286675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.286652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 21:06:08.286814 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.286724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 21:06:08.289308 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.289290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 21:06:08.289550 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.289536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 21:06:08.289635 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.289600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 21:06:08.289692 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.289671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 21:06:08.296107 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.296078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"]
Jul 29 21:06:08.303437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.303409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"]
Jul 29 21:06:08.369873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.369831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/113a3246-4685-4305-b2da-deff140bc87b-klusterlet-config\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.369873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.369873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.369914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skx7w\" (UniqueName: \"kubernetes.io/projected/113a3246-4685-4305-b2da-deff140bc87b-kube-api-access-skx7w\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.369944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/113a3246-4685-4305-b2da-deff140bc87b-tmp\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.369960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.370007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77c74f8-3525-4087-8d73-863510af2ac7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.370047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.370063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.370091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.370080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftck\" (UniqueName: \"kubernetes.io/projected/a77c74f8-3525-4087-8d73-863510af2ac7-kube-api-access-hftck\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.470894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.470864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx7w\" (UniqueName: \"kubernetes.io/projected/113a3246-4685-4305-b2da-deff140bc87b-kube-api-access-skx7w\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.470987 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.470903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/113a3246-4685-4305-b2da-deff140bc87b-tmp\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.470987 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.470931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77c74f8-3525-4087-8d73-863510af2ac7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471159 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftck\" (UniqueName: \"kubernetes.io/projected/a77c74f8-3525-4087-8d73-863510af2ac7-kube-api-access-hftck\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/113a3246-4685-4305-b2da-deff140bc87b-klusterlet-config\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.471398 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.471611 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/113a3246-4685-4305-b2da-deff140bc87b-tmp\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.471869 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.471832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77c74f8-3525-4087-8d73-863510af2ac7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.473732 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.473684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.474040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.474015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/113a3246-4685-4305-b2da-deff140bc87b-klusterlet-config\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.474126 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.474054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.474343 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.474322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.474405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.474327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77c74f8-3525-4087-8d73-863510af2ac7-ca\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.479046 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.479022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftck\" (UniqueName: \"kubernetes.io/projected/a77c74f8-3525-4087-8d73-863510af2ac7-kube-api-access-hftck\") pod \"cluster-proxy-proxy-agent-56b9fb94d6-65jrg\" (UID: \"a77c74f8-3525-4087-8d73-863510af2ac7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.479373 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.479350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skx7w\" (UniqueName: \"kubernetes.io/projected/113a3246-4685-4305-b2da-deff140bc87b-kube-api-access-skx7w\") pod \"klusterlet-addon-workmgr-666c896499-mpqjr\" (UID: \"113a3246-4685-4305-b2da-deff140bc87b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.593834 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.593736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:08.612564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.612520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:06:08.733166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.733132    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"]
Jul 29 21:06:08.737271 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:06:08.737240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113a3246_4685_4305_b2da_deff140bc87b.slice/crio-04777eb627733b50ef1210ca3be7c7a9fb318217cdd5738d37914f99e7854e7b WatchSource:0}: Error finding container 04777eb627733b50ef1210ca3be7c7a9fb318217cdd5738d37914f99e7854e7b: Status 404 returned error can't find the container with id 04777eb627733b50ef1210ca3be7c7a9fb318217cdd5738d37914f99e7854e7b
Jul 29 21:06:08.751440 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.751413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"]
Jul 29 21:06:08.755159 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:06:08.755133    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda77c74f8_3525_4087_8d73_863510af2ac7.slice/crio-80e4da4cbab71d70ebecc8822f7775d78db6a322c260a15ce74a22331e33b92a WatchSource:0}: Error finding container 80e4da4cbab71d70ebecc8822f7775d78db6a322c260a15ce74a22331e33b92a: Status 404 returned error can't find the container with id 80e4da4cbab71d70ebecc8822f7775d78db6a322c260a15ce74a22331e33b92a
Jul 29 21:06:08.769757 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.769725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr" event={"ID":"113a3246-4685-4305-b2da-deff140bc87b","Type":"ContainerStarted","Data":"04777eb627733b50ef1210ca3be7c7a9fb318217cdd5738d37914f99e7854e7b"}
Jul 29 21:06:08.770717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:08.770675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerStarted","Data":"80e4da4cbab71d70ebecc8822f7775d78db6a322c260a15ce74a22331e33b92a"}
Jul 29 21:06:11.706939 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:06:11.706893    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b8hsh" podUID="72d5e612-cb4c-4305-bfbf-931fa65818c7"
Jul 29 21:06:11.780450 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:11.780421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:06:12.196344 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:06:12.196289    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:06:12.789319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:12.789211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerStarted","Data":"85bae81e14c45e6d21d335cf5235e05f326d8656a978a9af304175afe7597b3f"}
Jul 29 21:06:12.789319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:12.789275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerStarted","Data":"0ab98e8c6941e381fd376001203a60e1179e7596b5f5d829562f91d86cd85a35"}
Jul 29 21:06:12.789319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:12.789294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerStarted","Data":"bf4e4c582ee9e63e68e67e380c6b57342e9a66ed18561b088bc8d7fce568a493"}
Jul 29 21:06:12.808097 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:12.808037    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" podStartSLOduration=1.536305198 podStartE2EDuration="4.808020362s" totalImagesPullingTime="3.271715164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:06:12.80655375 +0000 UTC m=+160.105031504" watchObservedRunningTime="2026-07-29 21:06:12.808020362 +0000 UTC m=+160.106498117"
Jul 29 21:06:15.798817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:15.798782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr" event={"ID":"113a3246-4685-4305-b2da-deff140bc87b","Type":"ContainerStarted","Data":"5f4aa6fe7d9cb81a5c4339296d1eab4dad56ac90d038bc8b086dc3fca01c970d"}
Jul 29 21:06:15.799379 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:15.798987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:15.800607 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:15.800587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:06:15.817673 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:15.815017    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr" podStartSLOduration=1.533444495 podStartE2EDuration="7.814967751s" totalImagesPullingTime="6.281523256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:06:15.814510559 +0000 UTC m=+163.112988315" watchObservedRunningTime="2026-07-29 21:06:15.814967751 +0000 UTC m=+163.113469982"
Jul 29 21:06:16.535805 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:16.535751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:06:16.535990 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:06:16.535878    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:06:16.535990 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:06:16.535933    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls podName:72d5e612-cb4c-4305-bfbf-931fa65818c7 nodeName:}" failed. No retries permitted until 2026-07-29 21:08:18.535918216 +0000 UTC m=+285.834395950 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls") pod "dns-default-b8hsh" (UID: "72d5e612-cb4c-4305-bfbf-931fa65818c7") : secret "dns-default-metrics-tls" not found
Jul 29 21:06:23.181983 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:06:23.181950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:07:43.319835 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.319797    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wqz9"]
Jul 29 21:07:43.321594 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.321576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.325116 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.325093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qz6qg\""
Jul 29 21:07:43.325266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.325249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 21:07:43.325472 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.325458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 21:07:43.325612 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.325597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 21:07:43.326383 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.326369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 21:07:43.337596 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.337572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wqz9"]
Jul 29 21:07:43.454796 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.454758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-crio-socket\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.454996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.454825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.454996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.454898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.454996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.454934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-data-volume\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.454996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.454965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdndq\" (UniqueName: \"kubernetes.io/projected/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-api-access-sdndq\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdndq\" (UniqueName: \"kubernetes.io/projected/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-api-access-sdndq\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-crio-socket\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-data-volume\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-crio-socket\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.556464 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:43.556405    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:07:43.556670 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:43.556407    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:43.556670 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:43.556483    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:44.056468564 +0000 UTC m=+251.354946310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : configmap "kube-rbac-proxy" not found
Jul 29 21:07:43.556670 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:43.556545    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:44.05652949 +0000 UTC m=+251.355007223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:43.556670 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.556592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-data-volume\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:43.565397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:43.565377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdndq\" (UniqueName: \"kubernetes.io/projected/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-api-access-sdndq\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:44.059261 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:44.059212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:44.059497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:44.059273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:44.059497 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:44.059353    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:07:44.059497 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:44.059386    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:44.059497 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:44.059423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:45.059400932 +0000 UTC m=+252.357878684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : configmap "kube-rbac-proxy" not found
Jul 29 21:07:44.059497 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:44.059442    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:45.059430115 +0000 UTC m=+252.357907869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:45.065899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:45.065858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:45.065914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:45.065941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066017    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066035    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066043    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066069    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:47.066056055 +0000 UTC m=+254.364533789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : configmap "kube-rbac-proxy" not found
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066107    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:47.066083903 +0000 UTC m=+254.364561636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:45.066328 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:45.066127    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs podName:b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2 nodeName:}" failed. No retries permitted until 2026-07-29 21:09:47.066117081 +0000 UTC m=+374.364594820 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs") pod "network-metrics-daemon-4tv28" (UID: "b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2") : secret "metrics-daemon-secret" not found
Jul 29 21:07:45.305495 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:45.305468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dn9wv_a07af5c6-3d90-4231-9f57-63f3566f8f62/dns-node-resolver/0.log"
Jul 29 21:07:46.504573 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:46.504547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t9785_e2a2e0e9-8711-4116-a953-cfd4d2b5c978/node-ca/0.log"
Jul 29 21:07:47.081765 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:47.081726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:47.081949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:47.081832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:47.081949 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:47.081852    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:07:47.081949 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:47.081914    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:51.081896948 +0000 UTC m=+258.380374686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : configmap "kube-rbac-proxy" not found
Jul 29 21:07:47.081949 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:47.081941    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:47.082119 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:47.081985    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:51.081974606 +0000 UTC m=+258.380452356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:47.104720 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:47.104671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jzv4d_b32cd7f0-fea6-4130-a158-a141dfff3207/serve-healthcheck-canary/0.log"
Jul 29 21:07:51.110014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:51.109980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:51.110014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:51.110021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:51.110462 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:51.110099    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:07:51.110462 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:51.110151    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:59.110137276 +0000 UTC m=+266.408615010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : configmap "kube-rbac-proxy" not found
Jul 29 21:07:51.110462 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:51.110156    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:51.110462 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:51.110221    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:07:59.110204143 +0000 UTC m=+266.408681878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:59.169895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:59.169802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:59.169895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:59.169851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:07:59.170292 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:59.169968    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:59.170292 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:07:59.170039    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls podName:4d2c23cf-45d3-40d4-97e8-4e906e60e4db nodeName:}" failed. No retries permitted until 2026-07-29 21:08:15.170022996 +0000 UTC m=+282.468500729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wqz9" (UID: "4d2c23cf-45d3-40d4-97e8-4e906e60e4db") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:07:59.170364 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:07:59.170303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:08:06.033094 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.033058    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b6b796dcf-znxdx"]
Jul 29 21:08:06.034812 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.034788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.038837 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.038816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 21:08:06.039140 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.039125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 21:08:06.039315 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.039300    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n9ncd\""
Jul 29 21:08:06.040012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.039994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 21:08:06.044476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.044459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 21:08:06.049038 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.049016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6b796dcf-znxdx"]
Jul 29 21:08:06.224882 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.224846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmdk6\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-kube-api-access-cmdk6\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.224894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-bound-sa-token\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.224938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-registry-tls\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.225047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-installation-pull-secrets\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.225100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-trusted-ca\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.225131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-registry-certificates\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.225171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80076396-c68c-474f-b03f-5de5cc07cd87-ca-trust-extracted\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.225232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.225203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-image-registry-private-configuration\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326534 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80076396-c68c-474f-b03f-5de5cc07cd87-ca-trust-extracted\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326534 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-image-registry-private-configuration\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmdk6\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-kube-api-access-cmdk6\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-bound-sa-token\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-registry-tls\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-installation-pull-secrets\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80076396-c68c-474f-b03f-5de5cc07cd87-ca-trust-extracted\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.326941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-trusted-ca\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.327086 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.326953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-registry-certificates\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.327796 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.327768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-registry-certificates\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.327796 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.327774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80076396-c68c-474f-b03f-5de5cc07cd87-trusted-ca\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.329258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.329231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-registry-tls\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.329258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.329258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-installation-pull-secrets\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.329426 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.329348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80076396-c68c-474f-b03f-5de5cc07cd87-image-registry-private-configuration\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.338236 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.338211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmdk6\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-kube-api-access-cmdk6\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.338236 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.338231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80076396-c68c-474f-b03f-5de5cc07cd87-bound-sa-token\") pod \"image-registry-6b6b796dcf-znxdx\" (UID: \"80076396-c68c-474f-b03f-5de5cc07cd87\") " pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.344147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.344127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:06.477443 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:06.477410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6b796dcf-znxdx"]
Jul 29 21:08:06.480547 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:08:06.480518    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80076396_c68c_474f_b03f_5de5cc07cd87.slice/crio-71df6044102ca9bde91a074101154fa7f6705abd90acd171bcf9eebc5d2113d2 WatchSource:0}: Error finding container 71df6044102ca9bde91a074101154fa7f6705abd90acd171bcf9eebc5d2113d2: Status 404 returned error can't find the container with id 71df6044102ca9bde91a074101154fa7f6705abd90acd171bcf9eebc5d2113d2
Jul 29 21:08:07.119279 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:07.119246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" event={"ID":"80076396-c68c-474f-b03f-5de5cc07cd87","Type":"ContainerStarted","Data":"81225ae21b6b95d5e3e60d73f5e54520fc8ff15ae7e25a811e3d04b50a1e3216"}
Jul 29 21:08:07.119721 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:07.119297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" event={"ID":"80076396-c68c-474f-b03f-5de5cc07cd87","Type":"ContainerStarted","Data":"71df6044102ca9bde91a074101154fa7f6705abd90acd171bcf9eebc5d2113d2"}
Jul 29 21:08:07.119721 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:07.119384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:07.140234 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:07.140176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" podStartSLOduration=1.140159625 podStartE2EDuration="1.140159625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:07.138533987 +0000 UTC m=+274.437011745" watchObservedRunningTime="2026-07-29 21:08:07.140159625 +0000 UTC m=+274.438637409"
Jul 29 21:08:14.781558 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:08:14.781502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b8hsh" podUID="72d5e612-cb4c-4305-bfbf-931fa65818c7"
Jul 29 21:08:15.142690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.142659    2581 generic.go:356] "Generic (PLEG): container finished" podID="113a3246-4685-4305-b2da-deff140bc87b" containerID="5f4aa6fe7d9cb81a5c4339296d1eab4dad56ac90d038bc8b086dc3fca01c970d" exitCode=1
Jul 29 21:08:15.142855 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.142728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr" event={"ID":"113a3246-4685-4305-b2da-deff140bc87b","Type":"ContainerDied","Data":"5f4aa6fe7d9cb81a5c4339296d1eab4dad56ac90d038bc8b086dc3fca01c970d"}
Jul 29 21:08:15.142855 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.142763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:15.143143 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.143129    2581 scope.go:117] "RemoveContainer" containerID="5f4aa6fe7d9cb81a5c4339296d1eab4dad56ac90d038bc8b086dc3fca01c970d"
Jul 29 21:08:15.194312 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.194278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:08:15.196997 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.196973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d2c23cf-45d3-40d4-97e8-4e906e60e4db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wqz9\" (UID: \"4d2c23cf-45d3-40d4-97e8-4e906e60e4db\") " pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:08:15.430878 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.430776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wqz9"
Jul 29 21:08:15.559988 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.559964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wqz9"]
Jul 29 21:08:15.561680 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:08:15.561654    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2c23cf_45d3_40d4_97e8_4e906e60e4db.slice/crio-d8dd9c47be2e3e2a5e38e881cafe67e2f9146b5981598e4a2131b01aede4d16c WatchSource:0}: Error finding container d8dd9c47be2e3e2a5e38e881cafe67e2f9146b5981598e4a2131b01aede4d16c: Status 404 returned error can't find the container with id d8dd9c47be2e3e2a5e38e881cafe67e2f9146b5981598e4a2131b01aede4d16c
Jul 29 21:08:15.799133 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:15.799039    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:08:16.149478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:16.149436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr" event={"ID":"113a3246-4685-4305-b2da-deff140bc87b","Type":"ContainerStarted","Data":"77f0194bcbe82fba3196a04994df7d62f0396c3ba1f9695438b5d2541992072f"}
Jul 29 21:08:16.150089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:16.150044    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:08:16.150909 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:16.150847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666c896499-mpqjr"
Jul 29 21:08:16.152363 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:16.152321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wqz9" event={"ID":"4d2c23cf-45d3-40d4-97e8-4e906e60e4db","Type":"ContainerStarted","Data":"d8dd9c47be2e3e2a5e38e881cafe67e2f9146b5981598e4a2131b01aede4d16c"}
Jul 29 21:08:18.619938 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:18.619894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:18.622898 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:18.622862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d5e612-cb4c-4305-bfbf-931fa65818c7-metrics-tls\") pod \"dns-default-b8hsh\" (UID: \"72d5e612-cb4c-4305-bfbf-931fa65818c7\") " pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:18.745536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:18.745509    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5hxhk\""
Jul 29 21:08:18.753442 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:18.753411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:20.492073 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:20.491999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8hsh"]
Jul 29 21:08:20.494582 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:08:20.494555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d5e612_cb4c_4305_bfbf_931fa65818c7.slice/crio-a7d84a8969b941b28eb95cec8f04972145e0417d98c21c05f31e5c59585dad5a WatchSource:0}: Error finding container a7d84a8969b941b28eb95cec8f04972145e0417d98c21c05f31e5c59585dad5a: Status 404 returned error can't find the container with id a7d84a8969b941b28eb95cec8f04972145e0417d98c21c05f31e5c59585dad5a
Jul 29 21:08:21.172338 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:21.172256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wqz9" event={"ID":"4d2c23cf-45d3-40d4-97e8-4e906e60e4db","Type":"ContainerStarted","Data":"e75aa9afb4e9463db3f0eb67af2cbc8c656844844a1116863abf4a61009ec090"}
Jul 29 21:08:21.174874 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:21.174841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8hsh" event={"ID":"72d5e612-cb4c-4305-bfbf-931fa65818c7","Type":"ContainerStarted","Data":"a7d84a8969b941b28eb95cec8f04972145e0417d98c21c05f31e5c59585dad5a"}
Jul 29 21:08:23.998654 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:23.998560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g2t7f"]
Jul 29 21:08:24.009008 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.008981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.014298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.014269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 21:08:24.014833 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.014809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 21:08:24.015026 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.014858    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 21:08:24.015355 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.015334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cjm7d\""
Jul 29 21:08:24.015510 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.015367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 21:08:24.015742 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.015727    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 21:08:24.015830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.015785    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 21:08:24.071711 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-root\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-metrics-client-ca\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-sys\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2bn\" (UniqueName: \"kubernetes.io/projected/b007e500-cf5e-428a-82af-9c9348d49d0a-kube-api-access-dk2bn\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.071895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-textfile\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.072177 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-wtmp\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.072177 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.071924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172609 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-wtmp\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172609 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-wtmp\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-root\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.172939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-metrics-client-ca\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-sys\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.172978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2bn\" (UniqueName: \"kubernetes.io/projected/b007e500-cf5e-428a-82af-9c9348d49d0a-kube-api-access-dk2bn\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-root\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-textfile\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173179 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:08:24.173124    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 21:08:24.173360 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:08:24.173185    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls podName:b007e500-cf5e-428a-82af-9c9348d49d0a nodeName:}" failed. No retries permitted until 2026-07-29 21:08:24.67316561 +0000 UTC m=+291.971643348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls") pod "node-exporter-g2t7f" (UID: "b007e500-cf5e-428a-82af-9c9348d49d0a") : secret "node-exporter-tls" not found
Jul 29 21:08:24.173421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b007e500-cf5e-428a-82af-9c9348d49d0a-sys\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-textfile\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.173748 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.173727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b007e500-cf5e-428a-82af-9c9348d49d0a-metrics-client-ca\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.175225 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.175198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.185756 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.185724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2bn\" (UniqueName: \"kubernetes.io/projected/b007e500-cf5e-428a-82af-9c9348d49d0a-kube-api-access-dk2bn\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.677406 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.677351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.679905 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.679875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b007e500-cf5e-428a-82af-9c9348d49d0a-node-exporter-tls\") pod \"node-exporter-g2t7f\" (UID: \"b007e500-cf5e-428a-82af-9c9348d49d0a\") " pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.919636 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:24.919601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2t7f"
Jul 29 21:08:24.930884 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:08:24.930854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007e500_cf5e_428a_82af_9c9348d49d0a.slice/crio-c036f3e54867d9096a2ac3a06cfaa9fceafc68921076b57542d33a19b97e43bd WatchSource:0}: Error finding container c036f3e54867d9096a2ac3a06cfaa9fceafc68921076b57542d33a19b97e43bd: Status 404 returned error can't find the container with id c036f3e54867d9096a2ac3a06cfaa9fceafc68921076b57542d33a19b97e43bd
Jul 29 21:08:25.191859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.191775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wqz9" event={"ID":"4d2c23cf-45d3-40d4-97e8-4e906e60e4db","Type":"ContainerStarted","Data":"b52f2fc1db92c66ae5c6477946a2058106ab640be4038f0f19e4f402063b4597"}
Jul 29 21:08:25.193835 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.193803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8hsh" event={"ID":"72d5e612-cb4c-4305-bfbf-931fa65818c7","Type":"ContainerStarted","Data":"9e161dd33d2c7137383c766257f58960ee3633e8c2048985877a4b5370dff7a0"}
Jul 29 21:08:25.194050 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.193863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8hsh" event={"ID":"72d5e612-cb4c-4305-bfbf-931fa65818c7","Type":"ContainerStarted","Data":"a42d55c2001262d9df192a09023d025950367b95d5ee16ad5c49ef8fa1b053d7"}
Jul 29 21:08:25.194050 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.193981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:25.195205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.195165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2t7f" event={"ID":"b007e500-cf5e-428a-82af-9c9348d49d0a","Type":"ContainerStarted","Data":"c036f3e54867d9096a2ac3a06cfaa9fceafc68921076b57542d33a19b97e43bd"}
Jul 29 21:08:25.216813 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:25.216745    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b8hsh" podStartSLOduration=253.456201053 podStartE2EDuration="4m17.216726628s" totalImagesPullingTime="3.760525575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:25.215506888 +0000 UTC m=+292.513984645" watchObservedRunningTime="2026-07-29 21:08:25.216726628 +0000 UTC m=+292.515204384"
Jul 29 21:08:26.183443 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:08:26.183352    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4tv28" podUID="b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2"
Jul 29 21:08:26.202069 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:26.202016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2t7f" event={"ID":"b007e500-cf5e-428a-82af-9c9348d49d0a","Type":"ContainerStarted","Data":"0151ba08785c8cb1bc820d599f1edb29378e8939d30751fa70264e675b297159"}
Jul 29 21:08:26.351004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:26.350955    2581 patch_prober.go:28] interesting pod/image-registry-6b6b796dcf-znxdx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 21:08:26.351202 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:26.351039    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" podUID="80076396-c68c-474f-b03f-5de5cc07cd87" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:08:27.208091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:27.208052    2581 generic.go:356] "Generic (PLEG): container finished" podID="b007e500-cf5e-428a-82af-9c9348d49d0a" containerID="0151ba08785c8cb1bc820d599f1edb29378e8939d30751fa70264e675b297159" exitCode=0
Jul 29 21:08:27.208610 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:27.208129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2t7f" event={"ID":"b007e500-cf5e-428a-82af-9c9348d49d0a","Type":"ContainerDied","Data":"0151ba08785c8cb1bc820d599f1edb29378e8939d30751fa70264e675b297159"}
Jul 29 21:08:28.126367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.126332    2581 patch_prober.go:28] interesting pod/image-registry-6b6b796dcf-znxdx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 21:08:28.126550 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.126386    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" podUID="80076396-c68c-474f-b03f-5de5cc07cd87" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:08:28.213827 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.213720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wqz9" event={"ID":"4d2c23cf-45d3-40d4-97e8-4e906e60e4db","Type":"ContainerStarted","Data":"20a88d3f8225646a7ad443131fe3323706807076c2295c429c9d702841d31e72"}
Jul 29 21:08:28.215824 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.215796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2t7f" event={"ID":"b007e500-cf5e-428a-82af-9c9348d49d0a","Type":"ContainerStarted","Data":"a32c4f0dad8365196685081051cf8f085b58320f85009979e156d8752645d9db"}
Jul 29 21:08:28.215937 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.215849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2t7f" event={"ID":"b007e500-cf5e-428a-82af-9c9348d49d0a","Type":"ContainerStarted","Data":"1250dff6147fa1d33cadf7f3fafb9311856975c753d9f75dce7726f0f5f45a2b"}
Jul 29 21:08:28.231671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.231625    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4wqz9" podStartSLOduration=32.914128621 podStartE2EDuration="45.231608082s" totalImagesPullingTime="12.317479461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:07:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:28.231117335 +0000 UTC m=+295.529595089" watchObservedRunningTime="2026-07-29 21:08:28.231608082 +0000 UTC m=+295.530085837"
Jul 29 21:08:28.249290 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:28.249238    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g2t7f" podStartSLOduration=4.044185964 podStartE2EDuration="5.249192809s" totalImagesPullingTime="1.117122705s" totalInitContainerRuntime="87.88414ms" isStatefulPod=true podCreationTimestamp="2026-07-29 21:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:28.248560364 +0000 UTC m=+295.547038118" watchObservedRunningTime="2026-07-29 21:08:28.249192809 +0000 UTC m=+295.547670618"
Jul 29 21:08:33.045339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:33.045306    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:08:33.048626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:33.048597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:08:33.052356 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:33.052331    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 21:08:35.204254 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:35.204224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8hsh"
Jul 29 21:08:36.348478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:36.348440    2581 patch_prober.go:28] interesting pod/image-registry-6b6b796dcf-znxdx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 21:08:36.348899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:36.348498    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx" podUID="80076396-c68c-474f-b03f-5de5cc07cd87" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:08:38.126745 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:38.126718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b6b796dcf-znxdx"
Jul 29 21:08:38.181111 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:38.181082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:08:48.614324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:48.614278    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" podUID="a77c74f8-3525-4087-8d73-863510af2ac7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 21:08:55.679207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.679164    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:08:55.682206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.682189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.684464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.684438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-64d8z\""
Jul 29 21:08:55.684612 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.684599    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 21:08:55.685425 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.685397    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 21:08:55.685425 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.685414    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 21:08:55.685574 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.685473    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 21:08:55.685574 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.685522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 21:08:55.685792 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.685778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 21:08:55.686550 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.686533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 21:08:55.690035 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.690013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 21:08:55.695258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.695226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:08:55.820073 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820258 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdgb\" (UniqueName: \"kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.820384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.820315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921319 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdgb\" (UniqueName: \"kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.921659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.921572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.922201 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.922177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.922337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.922311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.922406 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.922177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.922406 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.922394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.923908 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.923891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.924072 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.924056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.930116 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.930067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdgb\" (UniqueName: \"kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb\") pod \"console-886689cb8-v46g2\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") " pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:55.996961 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:55.996927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:08:56.125378 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:56.125345    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:08:56.128412 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:08:56.128381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e382ab5_04dd_42e8_af3a_123b0674783d.slice/crio-afa86bfb6c00cd1e649e93c7ef39e0d53eb2fbb25053e46a8ab81b0a5978de93 WatchSource:0}: Error finding container afa86bfb6c00cd1e649e93c7ef39e0d53eb2fbb25053e46a8ab81b0a5978de93: Status 404 returned error can't find the container with id afa86bfb6c00cd1e649e93c7ef39e0d53eb2fbb25053e46a8ab81b0a5978de93
Jul 29 21:08:56.130191 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:56.130176    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:08:56.304456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:56.304364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-886689cb8-v46g2" event={"ID":"6e382ab5-04dd-42e8-af3a-123b0674783d","Type":"ContainerStarted","Data":"afa86bfb6c00cd1e649e93c7ef39e0d53eb2fbb25053e46a8ab81b0a5978de93"}
Jul 29 21:08:58.614422 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:58.614375    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" podUID="a77c74f8-3525-4087-8d73-863510af2ac7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 21:08:59.316792 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:59.316688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-886689cb8-v46g2" event={"ID":"6e382ab5-04dd-42e8-af3a-123b0674783d","Type":"ContainerStarted","Data":"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"}
Jul 29 21:08:59.335070 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:08:59.335018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-886689cb8-v46g2" podStartSLOduration=1.631941804 podStartE2EDuration="4.334999205s" totalImagesPullingTime="2.703057401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:59.334401453 +0000 UTC m=+326.632879208" watchObservedRunningTime="2026-07-29 21:08:59.334999205 +0000 UTC m=+326.633476959"
Jul 29 21:09:05.997976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:05.997927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:09:05.998375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:05.998002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:09:06.002997 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:06.002973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:09:06.341963 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:06.341866    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:09:08.614306 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:08.614262    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" podUID="a77c74f8-3525-4087-8d73-863510af2ac7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 29 21:09:08.614798 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:08.614353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg"
Jul 29 21:09:08.615042 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:08.615007    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"85bae81e14c45e6d21d335cf5235e05f326d8656a978a9af304175afe7597b3f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 29 21:09:08.615122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:08.615069    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" podUID="a77c74f8-3525-4087-8d73-863510af2ac7" containerName="service-proxy" containerID="cri-o://85bae81e14c45e6d21d335cf5235e05f326d8656a978a9af304175afe7597b3f" gracePeriod=30
Jul 29 21:09:09.352135 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:09.352093    2581 generic.go:356] "Generic (PLEG): container finished" podID="a77c74f8-3525-4087-8d73-863510af2ac7" containerID="85bae81e14c45e6d21d335cf5235e05f326d8656a978a9af304175afe7597b3f" exitCode=2
Jul 29 21:09:09.352313 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:09.352157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerDied","Data":"85bae81e14c45e6d21d335cf5235e05f326d8656a978a9af304175afe7597b3f"}
Jul 29 21:09:09.352313 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:09.352211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b9fb94d6-65jrg" event={"ID":"a77c74f8-3525-4087-8d73-863510af2ac7","Type":"ContainerStarted","Data":"6fd2ea4d72462456c62819f5ccb0fee22f96d9122ec6ade2be00a361a6679df3"}
Jul 29 21:09:47.132311 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.132217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:09:47.134730 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.134691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2-metrics-certs\") pod \"network-metrics-daemon-4tv28\" (UID: \"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2\") " pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:09:47.184830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.184799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82zmz\""
Jul 29 21:09:47.192852 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.192824    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4tv28"
Jul 29 21:09:47.330719 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.330666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4tv28"]
Jul 29 21:09:47.333778 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:09:47.333747    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb669f1f7_e1f3_48ba_8d67_fb7f94bda5c2.slice/crio-df6f6dccbc28fdd7d63d878fa0cd89fc3ce18195a90768fdc57ce62658a321a4 WatchSource:0}: Error finding container df6f6dccbc28fdd7d63d878fa0cd89fc3ce18195a90768fdc57ce62658a321a4: Status 404 returned error can't find the container with id df6f6dccbc28fdd7d63d878fa0cd89fc3ce18195a90768fdc57ce62658a321a4
Jul 29 21:09:47.476301 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:47.476213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tv28" event={"ID":"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2","Type":"ContainerStarted","Data":"df6f6dccbc28fdd7d63d878fa0cd89fc3ce18195a90768fdc57ce62658a321a4"}
Jul 29 21:09:48.481839 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:48.481796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tv28" event={"ID":"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2","Type":"ContainerStarted","Data":"f5b436a3f3997f3e4e2bd78c7b3ba9f947949c9298dd67996599ff1abff4c007"}
Jul 29 21:09:49.486566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:49.486527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4tv28" event={"ID":"b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2","Type":"ContainerStarted","Data":"4f100c2fc83ebd26123290cf13bfec399836167497fdbedd12eb20a06e0e3102"}
Jul 29 21:09:49.502722 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:49.502645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4tv28" podStartSLOduration=374.49620521 podStartE2EDuration="6m15.502629629s" totalImagesPullingTime="1.006424419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:09:49.501662451 +0000 UTC m=+376.800140215" watchObservedRunningTime="2026-07-29 21:09:49.502629629 +0000 UTC m=+376.801107384"
Jul 29 21:09:52.101068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:09:52.101029    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:10:17.120468 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.120429    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-886689cb8-v46g2" podUID="6e382ab5-04dd-42e8-af3a-123b0674783d" containerName="console" containerID="cri-o://6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1" gracePeriod=15
Jul 29 21:10:17.353448 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.353423    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:10:17.451950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.451853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.451950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.451944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452330 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452330 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrdgb\" (UniqueName: \"kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb\") pod \"6e382ab5-04dd-42e8-af3a-123b0674783d\" (UID: \"6e382ab5-04dd-42e8-af3a-123b0674783d\") "
Jul 29 21:10:17.452439 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:10:17.452490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:10:17.452553 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca" (OuterVolumeSpecName: "service-ca") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:10:17.452553 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452540    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-oauth-serving-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.452757 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452557    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-trusted-ca-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.452757 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.452626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config" (OuterVolumeSpecName: "console-config") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:10:17.454379 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.454355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:10:17.454499 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.454477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:10:17.454499 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.454478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb" (OuterVolumeSpecName: "kube-api-access-rrdgb") pod "6e382ab5-04dd-42e8-af3a-123b0674783d" (UID: "6e382ab5-04dd-42e8-af3a-123b0674783d"). InnerVolumeSpecName "kube-api-access-rrdgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:10:17.553414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.553361    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-serving-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.553414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.553406    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.553414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.553416    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e382ab5-04dd-42e8-af3a-123b0674783d-console-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.553414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.553425    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdgb\" (UniqueName: \"kubernetes.io/projected/6e382ab5-04dd-42e8-af3a-123b0674783d-kube-api-access-rrdgb\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.553414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.553434    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e382ab5-04dd-42e8-af3a-123b0674783d-console-oauth-config\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:10:17.580982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.580954    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e382ab5-04dd-42e8-af3a-123b0674783d" containerID="6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1" exitCode=0
Jul 29 21:10:17.581139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.581003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-886689cb8-v46g2"
Jul 29 21:10:17.581139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.581021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-886689cb8-v46g2" event={"ID":"6e382ab5-04dd-42e8-af3a-123b0674783d","Type":"ContainerDied","Data":"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"}
Jul 29 21:10:17.581139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.581061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-886689cb8-v46g2" event={"ID":"6e382ab5-04dd-42e8-af3a-123b0674783d","Type":"ContainerDied","Data":"afa86bfb6c00cd1e649e93c7ef39e0d53eb2fbb25053e46a8ab81b0a5978de93"}
Jul 29 21:10:17.581139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.581080    2581 scope.go:117] "RemoveContainer" containerID="6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"
Jul 29 21:10:17.589944 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.589927    2581 scope.go:117] "RemoveContainer" containerID="6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"
Jul 29 21:10:17.590235 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:10:17.590213    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1\": container with ID starting with 6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1 not found: ID does not exist" containerID="6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"
Jul 29 21:10:17.590295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.590246    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1"} err="failed to get container status \"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1\": rpc error: code = NotFound desc = could not find container \"6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1\": container with ID starting with 6959a97caa7a808729f81e74ffdf6352112a2f0739ab12e741fa6da11a7801f1 not found: ID does not exist"
Jul 29 21:10:17.601178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.601153    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:10:17.605231 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:17.605210    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-886689cb8-v46g2"]
Jul 29 21:10:19.186084 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:19.186040    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e382ab5-04dd-42e8-af3a-123b0674783d" path="/var/lib/kubelet/pods/6e382ab5-04dd-42e8-af3a-123b0674783d/volumes"
Jul 29 21:10:46.999399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:46.999354    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"]
Jul 29 21:10:46.999799 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:46.999590    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e382ab5-04dd-42e8-af3a-123b0674783d" containerName="console"
Jul 29 21:10:46.999799 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:46.999602    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e382ab5-04dd-42e8-af3a-123b0674783d" containerName="console"
Jul 29 21:10:46.999799 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:46.999676    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e382ab5-04dd-42e8-af3a-123b0674783d" containerName="console"
Jul 29 21:10:47.003706 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.003675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.006349 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.006321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xvvln\""
Jul 29 21:10:47.006466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.006356    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 21:10:47.006466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.006321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 21:10:47.010455 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.010135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"]
Jul 29 21:10:47.056328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.056283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.056328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.056329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqpz\" (UniqueName: \"kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.056538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.056441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.157306 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.157269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqpz\" (UniqueName: \"kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.157477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.157325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.157477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.157359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.157718 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.157680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.157777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.157759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.166245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.166212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqpz\" (UniqueName: \"kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.313022 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.312936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:10:47.442270 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.442235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"]
Jul 29 21:10:47.445378 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:10:47.445352    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd23ca8_3df2_490b_90fc_71f2a53ea925.slice/crio-82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2 WatchSource:0}: Error finding container 82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2: Status 404 returned error can't find the container with id 82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2
Jul 29 21:10:47.684598 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:47.684565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq" event={"ID":"edd23ca8-3df2-490b-90fc-71f2a53ea925","Type":"ContainerStarted","Data":"82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2"}
Jul 29 21:10:55.714309 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:55.714273    2581 generic.go:356] "Generic (PLEG): container finished" podID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerID="23a0b3f03f411163f170cbfc342599bac66a9612827832eadfca495391965e24" exitCode=0
Jul 29 21:10:55.714885 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:55.714348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq" event={"ID":"edd23ca8-3df2-490b-90fc-71f2a53ea925","Type":"ContainerDied","Data":"23a0b3f03f411163f170cbfc342599bac66a9612827832eadfca495391965e24"}
Jul 29 21:10:58.727404 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:58.727366    2581 generic.go:356] "Generic (PLEG): container finished" podID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerID="82239b5c84228bd83613a66ead8a7ce15a6907a7e8f9eb37323c5f68a232c37a" exitCode=0
Jul 29 21:10:58.727839 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:10:58.727452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq" event={"ID":"edd23ca8-3df2-490b-90fc-71f2a53ea925","Type":"ContainerDied","Data":"82239b5c84228bd83613a66ead8a7ce15a6907a7e8f9eb37323c5f68a232c37a"}
Jul 29 21:11:06.759472 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:06.759436    2581 generic.go:356] "Generic (PLEG): container finished" podID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerID="1ed3eaaf3d3553287994cf5c4cd5ce8214bb140af5ba1929dadf7cb9a2f9678d" exitCode=0
Jul 29 21:11:06.759972 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:06.759509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq" event={"ID":"edd23ca8-3df2-490b-90fc-71f2a53ea925","Type":"ContainerDied","Data":"1ed3eaaf3d3553287994cf5c4cd5ce8214bb140af5ba1929dadf7cb9a2f9678d"}
Jul 29 21:11:07.894165 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:07.894136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:11:08.024643 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.024549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhqpz\" (UniqueName: \"kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz\") pod \"edd23ca8-3df2-490b-90fc-71f2a53ea925\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") "
Jul 29 21:11:08.024643 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.024596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util\") pod \"edd23ca8-3df2-490b-90fc-71f2a53ea925\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") "
Jul 29 21:11:08.024643 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.024638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle\") pod \"edd23ca8-3df2-490b-90fc-71f2a53ea925\" (UID: \"edd23ca8-3df2-490b-90fc-71f2a53ea925\") "
Jul 29 21:11:08.025344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.025304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle" (OuterVolumeSpecName: "bundle") pod "edd23ca8-3df2-490b-90fc-71f2a53ea925" (UID: "edd23ca8-3df2-490b-90fc-71f2a53ea925"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:08.026980 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.026946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz" (OuterVolumeSpecName: "kube-api-access-lhqpz") pod "edd23ca8-3df2-490b-90fc-71f2a53ea925" (UID: "edd23ca8-3df2-490b-90fc-71f2a53ea925"). InnerVolumeSpecName "kube-api-access-lhqpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:11:08.029315 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.029291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util" (OuterVolumeSpecName: "util") pod "edd23ca8-3df2-490b-90fc-71f2a53ea925" (UID: "edd23ca8-3df2-490b-90fc-71f2a53ea925"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:08.125830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.125797    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:08.125830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.125820    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhqpz\" (UniqueName: \"kubernetes.io/projected/edd23ca8-3df2-490b-90fc-71f2a53ea925-kube-api-access-lhqpz\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:08.125830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.125833    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edd23ca8-3df2-490b-90fc-71f2a53ea925-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:08.767822 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.767786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq" event={"ID":"edd23ca8-3df2-490b-90fc-71f2a53ea925","Type":"ContainerDied","Data":"82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2"}
Jul 29 21:11:08.768014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.767827    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mm5tq"
Jul 29 21:11:08.768014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:08.767833    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a486db0c5a29899a66b305c9dd42ddd8619f918490624cc308e436257b96c2"
Jul 29 21:11:19.414675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414637    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"]
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="pull"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414921    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="pull"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414928    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="extract"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="extract"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414942    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="util"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414948    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="util"
Jul 29 21:11:19.415098 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.414997    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd23ca8-3df2-490b-90fc-71f2a53ea925" containerName="extract"
Jul 29 21:11:19.418245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.418223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.420668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.420648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xvvln\""
Jul 29 21:11:19.420795 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.420646    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 21:11:19.421539 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.421523    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 21:11:19.426841 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.426815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"]
Jul 29 21:11:19.511229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.511181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.511229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.511233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.511466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.511310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2sj\" (UniqueName: \"kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.612482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.612443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.612482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.612486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.612689 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.612533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2sj\" (UniqueName: \"kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.612868 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.612850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.612932 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.612911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.621513 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.621473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2sj\" (UniqueName: \"kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.727549 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.727453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:19.858932 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:19.858900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"]
Jul 29 21:11:19.862153 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:11:19.862125    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2568b3_09ef_4c57_b790_e5c29d6757f8.slice/crio-5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc WatchSource:0}: Error finding container 5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc: Status 404 returned error can't find the container with id 5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc
Jul 29 21:11:20.809610 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:20.809579    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerID="795575d6f6d83874ebe6eac324ab7a1d7706ec7c79f0079436602bf589af64e7" exitCode=0
Jul 29 21:11:20.810018 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:20.809661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4" event={"ID":"ee2568b3-09ef-4c57-b790-e5c29d6757f8","Type":"ContainerDied","Data":"795575d6f6d83874ebe6eac324ab7a1d7706ec7c79f0079436602bf589af64e7"}
Jul 29 21:11:20.810018 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:20.809723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4" event={"ID":"ee2568b3-09ef-4c57-b790-e5c29d6757f8","Type":"ContainerStarted","Data":"5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc"}
Jul 29 21:11:23.822290 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:23.822253    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerID="b9d643ccbe8e512b6356b44de51b90f70460b7cb96a532a1d32151a42b4640ea" exitCode=0
Jul 29 21:11:23.822795 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:23.822322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4" event={"ID":"ee2568b3-09ef-4c57-b790-e5c29d6757f8","Type":"ContainerDied","Data":"b9d643ccbe8e512b6356b44de51b90f70460b7cb96a532a1d32151a42b4640ea"}
Jul 29 21:11:24.828310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:24.828276    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerID="9c3577379282c7b9eb89e4febacbbfd85536d2a29161d52bf45d36fab000b30e" exitCode=0
Jul 29 21:11:24.828733 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:24.828358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4" event={"ID":"ee2568b3-09ef-4c57-b790-e5c29d6757f8","Type":"ContainerDied","Data":"9c3577379282c7b9eb89e4febacbbfd85536d2a29161d52bf45d36fab000b30e"}
Jul 29 21:11:25.951032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:25.951006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:26.062957 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.062916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q2sj\" (UniqueName: \"kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj\") pod \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") "
Jul 29 21:11:26.062957 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.062959    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle\") pod \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") "
Jul 29 21:11:26.063218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.063027    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util\") pod \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\" (UID: \"ee2568b3-09ef-4c57-b790-e5c29d6757f8\") "
Jul 29 21:11:26.063380 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.063349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle" (OuterVolumeSpecName: "bundle") pod "ee2568b3-09ef-4c57-b790-e5c29d6757f8" (UID: "ee2568b3-09ef-4c57-b790-e5c29d6757f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:26.065362 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.065336    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj" (OuterVolumeSpecName: "kube-api-access-2q2sj") pod "ee2568b3-09ef-4c57-b790-e5c29d6757f8" (UID: "ee2568b3-09ef-4c57-b790-e5c29d6757f8"). InnerVolumeSpecName "kube-api-access-2q2sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:11:26.067718 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.067668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util" (OuterVolumeSpecName: "util") pod "ee2568b3-09ef-4c57-b790-e5c29d6757f8" (UID: "ee2568b3-09ef-4c57-b790-e5c29d6757f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:26.163732 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.163664    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:26.163732 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.163724    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2sj\" (UniqueName: \"kubernetes.io/projected/ee2568b3-09ef-4c57-b790-e5c29d6757f8-kube-api-access-2q2sj\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:26.163732 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.163735    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee2568b3-09ef-4c57-b790-e5c29d6757f8-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:26.836877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.836852    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4"
Jul 29 21:11:26.837060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.836844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg8nt4" event={"ID":"ee2568b3-09ef-4c57-b790-e5c29d6757f8","Type":"ContainerDied","Data":"5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc"}
Jul 29 21:11:26.837060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:26.836971    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a305734b028d69e4552b4d47b1392a1e9283223d268614c202212ea63b412fc"
Jul 29 21:11:39.675688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675649    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"]
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675914    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="pull"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="pull"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675940    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="extract"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="extract"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675958    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="util"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.675965    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="util"
Jul 29 21:11:39.676144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.676004    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2568b3-09ef-4c57-b790-e5c29d6757f8" containerName="extract"
Jul 29 21:11:39.696618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.696581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"]
Jul 29 21:11:39.696795 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.696711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.699471 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.699446    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 21:11:39.699471 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.699460    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 21:11:39.700141 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.700126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xvvln\""
Jul 29 21:11:39.869712 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.869677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.869874 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.869726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.869874 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.869803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvrm\" (UniqueName: \"kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.970257 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.970173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvrm\" (UniqueName: \"kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.970257 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.970235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.970257 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.970259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.970602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.970587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.970757 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.970738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:39.979498 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:39.979473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvrm\" (UniqueName: \"kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:40.006375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:40.006343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:40.152241 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:40.152202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"]
Jul 29 21:11:40.155044 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:11:40.155011    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89347125_2d89_45bf_8fd9_e5c4c446984e.slice/crio-2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3 WatchSource:0}: Error finding container 2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3: Status 404 returned error can't find the container with id 2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3
Jul 29 21:11:40.888945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:40.888908    2581 generic.go:356] "Generic (PLEG): container finished" podID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerID="882608a4e2485567e9014b0b1e69f0652d1f0a9a8a0aa9cb92301ebe90db2202" exitCode=0
Jul 29 21:11:40.889391 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:40.888974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw" event={"ID":"89347125-2d89-45bf-8fd9-e5c4c446984e","Type":"ContainerDied","Data":"882608a4e2485567e9014b0b1e69f0652d1f0a9a8a0aa9cb92301ebe90db2202"}
Jul 29 21:11:40.889391 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:40.889029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw" event={"ID":"89347125-2d89-45bf-8fd9-e5c4c446984e","Type":"ContainerStarted","Data":"2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3"}
Jul 29 21:11:42.898470 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:42.898438    2581 generic.go:356] "Generic (PLEG): container finished" podID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerID="ff95ff4d957f6ab4ae5ba3ebcfa0aa2257853bb83934eba5403e23d7a77a2dda" exitCode=0
Jul 29 21:11:42.898858 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:42.898526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw" event={"ID":"89347125-2d89-45bf-8fd9-e5c4c446984e","Type":"ContainerDied","Data":"ff95ff4d957f6ab4ae5ba3ebcfa0aa2257853bb83934eba5403e23d7a77a2dda"}
Jul 29 21:11:43.904238 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:43.904201    2581 generic.go:356] "Generic (PLEG): container finished" podID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerID="66ea85110e5abeda7efa070e0c547eaae90faafb8143c2a41fc0c462fe9af8aa" exitCode=0
Jul 29 21:11:43.904616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:43.904286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw" event={"ID":"89347125-2d89-45bf-8fd9-e5c4c446984e","Type":"ContainerDied","Data":"66ea85110e5abeda7efa070e0c547eaae90faafb8143c2a41fc0c462fe9af8aa"}
Jul 29 21:11:45.033943 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.033921    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:45.108247 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.108208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util\") pod \"89347125-2d89-45bf-8fd9-e5c4c446984e\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") "
Jul 29 21:11:45.108426 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.108284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlvrm\" (UniqueName: \"kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm\") pod \"89347125-2d89-45bf-8fd9-e5c4c446984e\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") "
Jul 29 21:11:45.108426 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.108326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle\") pod \"89347125-2d89-45bf-8fd9-e5c4c446984e\" (UID: \"89347125-2d89-45bf-8fd9-e5c4c446984e\") "
Jul 29 21:11:45.109137 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.109099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle" (OuterVolumeSpecName: "bundle") pod "89347125-2d89-45bf-8fd9-e5c4c446984e" (UID: "89347125-2d89-45bf-8fd9-e5c4c446984e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:45.110463 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.110437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm" (OuterVolumeSpecName: "kube-api-access-wlvrm") pod "89347125-2d89-45bf-8fd9-e5c4c446984e" (UID: "89347125-2d89-45bf-8fd9-e5c4c446984e"). InnerVolumeSpecName "kube-api-access-wlvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:11:45.116338 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.116306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util" (OuterVolumeSpecName: "util") pod "89347125-2d89-45bf-8fd9-e5c4c446984e" (UID: "89347125-2d89-45bf-8fd9-e5c4c446984e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:11:45.209173 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.209136    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:45.209173 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.209170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvrm\" (UniqueName: \"kubernetes.io/projected/89347125-2d89-45bf-8fd9-e5c4c446984e-kube-api-access-wlvrm\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:45.209173 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.209181    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89347125-2d89-45bf-8fd9-e5c4c446984e-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:11:45.913316 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.913281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw" event={"ID":"89347125-2d89-45bf-8fd9-e5c4c446984e","Type":"ContainerDied","Data":"2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3"}
Jul 29 21:11:45.913316 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.913314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl98sw"
Jul 29 21:11:45.913530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:45.913333    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae19a192caeef3907250aad9a644ef57cf2b2787108e925d0573adf9546d0b3"
Jul 29 21:11:56.982199 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"]
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982470    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="pull"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="pull"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982503    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="util"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982511    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="util"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982534    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="extract"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982542    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="extract"
Jul 29 21:11:56.982717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.982594    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89347125-2d89-45bf-8fd9-e5c4c446984e" containerName="extract"
Jul 29 21:11:56.984401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.984379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:56.987139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.987111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 21:11:56.987679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.987323    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 21:11:56.987679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.987391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9mx9\""
Jul 29 21:11:56.987679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.987579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 21:11:56.988297 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:56.988279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 21:11:57.003642 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.003610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"]
Jul 29 21:11:57.094131 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.094093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-manifests\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.094291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.094137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.094291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.094201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdgh\" (UniqueName: \"kubernetes.io/projected/1440618b-50cd-4d74-a105-c30046991ae0-kube-api-access-qvdgh\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.094291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.094262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.094291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.094281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-tmp\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.195501 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.195464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.195683 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.195529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdgh\" (UniqueName: \"kubernetes.io/projected/1440618b-50cd-4d74-a105-c30046991ae0-kube-api-access-qvdgh\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.195887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.195851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.196032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.195904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-tmp\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.196032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.195939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-manifests\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.196274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.196254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-manifests\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.198131 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.198103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1440618b-50cd-4d74-a105-c30046991ae0-tmp\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.198131 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.198123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.198444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.198422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1440618b-50cd-4d74-a105-c30046991ae0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.204384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.204362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdgh\" (UniqueName: \"kubernetes.io/projected/1440618b-50cd-4d74-a105-c30046991ae0-kube-api-access-qvdgh\") pod \"opendatahub-operator-controller-manager-8446ffcd9c-cr54w\" (UID: \"1440618b-50cd-4d74-a105-c30046991ae0\") " pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.295495 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.295388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:11:57.433853 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.433805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"]
Jul 29 21:11:57.438984 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:11:57.438947    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1440618b_50cd_4d74_a105_c30046991ae0.slice/crio-16bbc1268b928144338b81f972b6be45385d0bd2d421957addf6cde4b24caf9b WatchSource:0}: Error finding container 16bbc1268b928144338b81f972b6be45385d0bd2d421957addf6cde4b24caf9b: Status 404 returned error can't find the container with id 16bbc1268b928144338b81f972b6be45385d0bd2d421957addf6cde4b24caf9b
Jul 29 21:11:57.954804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:11:57.954767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w" event={"ID":"1440618b-50cd-4d74-a105-c30046991ae0","Type":"ContainerStarted","Data":"16bbc1268b928144338b81f972b6be45385d0bd2d421957addf6cde4b24caf9b"}
Jul 29 21:12:00.968991 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:00.968952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w" event={"ID":"1440618b-50cd-4d74-a105-c30046991ae0","Type":"ContainerStarted","Data":"9db683612f8ca3738ddcaf05b2879d022b290a1b7576ff5a277af3f5f9007a36"}
Jul 29 21:12:01.973639 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:01.973536    2581 generic.go:356] "Generic (PLEG): container finished" podID="1440618b-50cd-4d74-a105-c30046991ae0" containerID="9db683612f8ca3738ddcaf05b2879d022b290a1b7576ff5a277af3f5f9007a36" exitCode=0
Jul 29 21:12:01.973639 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:01.973625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w" event={"ID":"1440618b-50cd-4d74-a105-c30046991ae0","Type":"ContainerDied","Data":"9db683612f8ca3738ddcaf05b2879d022b290a1b7576ff5a277af3f5f9007a36"}
Jul 29 21:12:02.980920 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:02.980884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w" event={"ID":"1440618b-50cd-4d74-a105-c30046991ae0","Type":"ContainerStarted","Data":"1264489d1da24ffa139f4b5215d4b8b6ab6c0b64529af71b733285bb7dce20ee"}
Jul 29 21:12:02.981400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:02.981035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:12:03.002189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:03.002140    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w" podStartSLOduration=2.494139301 podStartE2EDuration="7.002123497s" totalImagesPullingTime="3.440528506s" totalInitContainerRuntime="1.06745569s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:11:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:12:02.999758531 +0000 UTC m=+510.298236283" watchObservedRunningTime="2026-07-29 21:12:03.002123497 +0000 UTC m=+510.300601252"
Jul 29 21:12:13.990095 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:13.990054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8446ffcd9c-cr54w"
Jul 29 21:12:16.855068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.855035    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"]
Jul 29 21:12:16.859615 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.859591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:16.862298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.862271    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 21:12:16.862437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.862274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 21:12:16.863334 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.863314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kczcl\""
Jul 29 21:12:16.863438 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.863317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 21:12:16.863438 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.863319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 21:12:16.879324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.879291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"]
Jul 29 21:12:16.945168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.945133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dbj\" (UniqueName: \"kubernetes.io/projected/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-kube-api-access-p9dbj\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:16.945168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.945173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:16.945361 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:16.945212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tmp\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.031402 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.031370    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mxbnj"]
Jul 29 21:12:17.033569 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.033551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.035917 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.035891    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 29 21:12:17.036202 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.036185    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-29xjw\""
Jul 29 21:12:17.046445 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.046409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.046597 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.046478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tmp\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.046597 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.046539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dbj\" (UniqueName: \"kubernetes.io/projected/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-kube-api-access-p9dbj\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.046597 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.046574    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 29 21:12:17.046786 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.046658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs podName:11c7faca-78cb-4dcc-bfe1-23b0dbb40376 nodeName:}" failed. No retries permitted until 2026-07-29 21:12:17.546635401 +0000 UTC m=+524.845113134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs") pod "kube-auth-proxy-75d94d9cdd-4kfsc" (UID: "11c7faca-78cb-4dcc-bfe1-23b0dbb40376") : secret "kube-auth-proxy-tls" not found
Jul 29 21:12:17.048911 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.048889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tmp\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.051757 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.051732    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mxbnj"]
Jul 29 21:12:17.059876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.059846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dbj\" (UniqueName: \"kubernetes.io/projected/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-kube-api-access-p9dbj\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.147505 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.147410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.147505 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.147492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqrc\" (UniqueName: \"kubernetes.io/projected/f9fe751a-da54-4097-8dea-d483673e6a62-kube-api-access-ttqrc\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.247874 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.247835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqrc\" (UniqueName: \"kubernetes.io/projected/f9fe751a-da54-4097-8dea-d483673e6a62-kube-api-access-ttqrc\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.248075 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.247909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.248075 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.248053    2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 21:12:17.248206 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.248136    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert podName:f9fe751a-da54-4097-8dea-d483673e6a62 nodeName:}" failed. No retries permitted until 2026-07-29 21:12:17.748114684 +0000 UTC m=+525.046592418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert") pod "odh-model-controller-7998c67dc4-mxbnj" (UID: "f9fe751a-da54-4097-8dea-d483673e6a62") : secret "odh-model-controller-webhook-cert" not found
Jul 29 21:12:17.263894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.263856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqrc\" (UniqueName: \"kubernetes.io/projected/f9fe751a-da54-4097-8dea-d483673e6a62-kube-api-access-ttqrc\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.551058 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.550970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.553616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.553584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11c7faca-78cb-4dcc-bfe1-23b0dbb40376-tls-certs\") pod \"kube-auth-proxy-75d94d9cdd-4kfsc\" (UID: \"11c7faca-78cb-4dcc-bfe1-23b0dbb40376\") " pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.751887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.751847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:17.752053 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.751964    2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 21:12:17.752053 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:17.752019    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert podName:f9fe751a-da54-4097-8dea-d483673e6a62 nodeName:}" failed. No retries permitted until 2026-07-29 21:12:18.752005076 +0000 UTC m=+526.050482808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert") pod "odh-model-controller-7998c67dc4-mxbnj" (UID: "f9fe751a-da54-4097-8dea-d483673e6a62") : secret "odh-model-controller-webhook-cert" not found
Jul 29 21:12:17.769954 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.769913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"
Jul 29 21:12:17.911416 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.911381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc"]
Jul 29 21:12:17.914081 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:12:17.914016    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c7faca_78cb_4dcc_bfe1_23b0dbb40376.slice/crio-0f5a19acb329d813164d60bb25e432438e0cbe9ed9309f66cbf804d80e69ea43 WatchSource:0}: Error finding container 0f5a19acb329d813164d60bb25e432438e0cbe9ed9309f66cbf804d80e69ea43: Status 404 returned error can't find the container with id 0f5a19acb329d813164d60bb25e432438e0cbe9ed9309f66cbf804d80e69ea43
Jul 29 21:12:17.973624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.973587    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zzq88"]
Jul 29 21:12:17.976212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.976191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:17.991424 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.991396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 29 21:12:17.994021 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:17.993993    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zzq88"]
Jul 29 21:12:18.037536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.037501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc" event={"ID":"11c7faca-78cb-4dcc-bfe1-23b0dbb40376","Type":"ContainerStarted","Data":"0f5a19acb329d813164d60bb25e432438e0cbe9ed9309f66cbf804d80e69ea43"}
Jul 29 21:12:18.054992 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.054953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.055161 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.055026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdv2\" (UniqueName: \"kubernetes.io/projected/f98a9db3-fe2d-4568-b811-aa1d76890856-kube-api-access-5bdv2\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.155618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.155578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdv2\" (UniqueName: \"kubernetes.io/projected/f98a9db3-fe2d-4568-b811-aa1d76890856-kube-api-access-5bdv2\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.155843 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.155681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.155907 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:18.155843    2581 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 29 21:12:18.155958 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:18.155923    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs podName:f98a9db3-fe2d-4568-b811-aa1d76890856 nodeName:}" failed. No retries permitted until 2026-07-29 21:12:18.655901844 +0000 UTC m=+525.954379582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs") pod "model-serving-api-54d89bd5fc-zzq88" (UID: "f98a9db3-fe2d-4568-b811-aa1d76890856") : secret "model-serving-api-tls" not found
Jul 29 21:12:18.164966 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.164936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdv2\" (UniqueName: \"kubernetes.io/projected/f98a9db3-fe2d-4568-b811-aa1d76890856-kube-api-access-5bdv2\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.196664 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.196624    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:18.198646 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.198623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.200998 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.200967    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 21:12:18.201148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.201098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 21:12:18.201562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.201544    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 21:12:18.201562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.201557    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jp8s2\""
Jul 29 21:12:18.201756 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.201730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 21:12:18.213105 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.213075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:18.256229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256681 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgcs9\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.256681 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.256502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357585 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357585 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgcs9\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.357872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.358066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.357886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.359062 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.359017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.360517 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.360493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.361121 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.361101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.361268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.361247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.361331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.361247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.365596 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.365574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgcs9\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.366344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.366321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-v48dn\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.508941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.508851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:18.660984 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.660945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.664639 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.664446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98a9db3-fe2d-4568-b811-aa1d76890856-tls-certs\") pod \"model-serving-api-54d89bd5fc-zzq88\" (UID: \"f98a9db3-fe2d-4568-b811-aa1d76890856\") " pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:18.667032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.667009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:18.668995 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:12:18.668956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81839d31_2fd1_4365_8295_99e684928602.slice/crio-1d0191f636c284e4cba5d221d1ddb30b7747be894f96e927fe80057426e32a1a WatchSource:0}: Error finding container 1d0191f636c284e4cba5d221d1ddb30b7747be894f96e927fe80057426e32a1a: Status 404 returned error can't find the container with id 1d0191f636c284e4cba5d221d1ddb30b7747be894f96e927fe80057426e32a1a
Jul 29 21:12:18.761546 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.761451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:18.764461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.764432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9fe751a-da54-4097-8dea-d483673e6a62-cert\") pod \"odh-model-controller-7998c67dc4-mxbnj\" (UID: \"f9fe751a-da54-4097-8dea-d483673e6a62\") " pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:18.843142 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.843108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:18.886654 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:18.886501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:19.004470 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:19.004363    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mxbnj"]
Jul 29 21:12:19.044375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:19.044339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" event={"ID":"81839d31-2fd1-4365-8295-99e684928602","Type":"ContainerStarted","Data":"1d0191f636c284e4cba5d221d1ddb30b7747be894f96e927fe80057426e32a1a"}
Jul 29 21:12:19.045428 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:19.045402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj" event={"ID":"f9fe751a-da54-4097-8dea-d483673e6a62","Type":"ContainerStarted","Data":"2fa9af5d3c9d4000051f8237efe7b97ce96d08d8e51320bb7dd0dc32a69de763"}
Jul 29 21:12:19.059197 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:19.059091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zzq88"]
Jul 29 21:12:19.062387 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:12:19.062347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98a9db3_fe2d_4568_b811_aa1d76890856.slice/crio-0113092264d90e039a63f9545b7b913bf3dc1bf5a6be00ee551fe3cbc12008e8 WatchSource:0}: Error finding container 0113092264d90e039a63f9545b7b913bf3dc1bf5a6be00ee551fe3cbc12008e8: Status 404 returned error can't find the container with id 0113092264d90e039a63f9545b7b913bf3dc1bf5a6be00ee551fe3cbc12008e8
Jul 29 21:12:20.061452 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:20.061373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88" event={"ID":"f98a9db3-fe2d-4568-b811-aa1d76890856","Type":"ContainerStarted","Data":"0113092264d90e039a63f9545b7b913bf3dc1bf5a6be00ee551fe3cbc12008e8"}
Jul 29 21:12:23.974326 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:23.974278    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 29 21:12:23.974608 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:23.974359    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 29 21:12:24.084583 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.084528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88" event={"ID":"f98a9db3-fe2d-4568-b811-aa1d76890856","Type":"ContainerStarted","Data":"1cf7fb30fdcf0ca615ef1d4cebf3bc8e5d8b7f59171446a50641176fa32bf1eb"}
Jul 29 21:12:24.084882 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.084848    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:12:24.086587 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.086530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj" event={"ID":"f9fe751a-da54-4097-8dea-d483673e6a62","Type":"ContainerStarted","Data":"aaa8abff6b4c6a72a05d9ee549ad62e5fa17b9e75ec9ed01051240abbec8486e"}
Jul 29 21:12:24.086773 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.086650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:24.088602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.088577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" event={"ID":"81839d31-2fd1-4365-8295-99e684928602","Type":"ContainerStarted","Data":"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"}
Jul 29 21:12:24.088745 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.088683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:24.090400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.090373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc" event={"ID":"11c7faca-78cb-4dcc-bfe1-23b0dbb40376","Type":"ContainerStarted","Data":"64db7fd010937372cd37b14aac7d87ab217d2ee23e506b034668f44eeb3cd379"}
Jul 29 21:12:24.106979 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.106919    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88" podStartSLOduration=2.2043956 podStartE2EDuration="7.106897807s" totalImagesPullingTime="4.902502207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:12:24.105583713 +0000 UTC m=+531.404061467" watchObservedRunningTime="2026-07-29 21:12:24.106897807 +0000 UTC m=+531.405375563"
Jul 29 21:12:24.124289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.124228    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj" podStartSLOduration=2.160842269 podStartE2EDuration="7.124212609s" totalImagesPullingTime="4.96337034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:12:24.122811834 +0000 UTC m=+531.421289611" watchObservedRunningTime="2026-07-29 21:12:24.124212609 +0000 UTC m=+531.422690363"
Jul 29 21:12:24.139622 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.139373    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75d94d9cdd-4kfsc" podStartSLOduration=2.088406521 podStartE2EDuration="8.139351202s" totalImagesPullingTime="6.050944681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:12:24.138976445 +0000 UTC m=+531.437454199" watchObservedRunningTime="2026-07-29 21:12:24.139351202 +0000 UTC m=+531.437828957"
Jul 29 21:12:24.165744 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:24.164943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" podStartSLOduration=0.862715285 podStartE2EDuration="6.164920906s" totalImagesPullingTime="5.302205621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:12:24.163812499 +0000 UTC m=+531.462290255" watchObservedRunningTime="2026-07-29 21:12:24.164920906 +0000 UTC m=+531.463398662"
Jul 29 21:12:25.097749 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:25.097718    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-v48dn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:12:25.098102 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:25.097771    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" podUID="81839d31-2fd1-4365-8295-99e684928602" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:12:28.096772 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:28.096743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:29.792964 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:29.792928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:29.793382 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:29.793134    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" podUID="81839d31-2fd1-4365-8295-99e684928602" containerName="discovery" containerID="cri-o://d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b" gracePeriod=30
Jul 29 21:12:30.041323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.041291    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:30.118779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.118740    2581 generic.go:356] "Generic (PLEG): container finished" podID="81839d31-2fd1-4365-8295-99e684928602" containerID="d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b" exitCode=0
Jul 29 21:12:30.118958 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.118792    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"
Jul 29 21:12:30.118958 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.118817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" event={"ID":"81839d31-2fd1-4365-8295-99e684928602","Type":"ContainerDied","Data":"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"}
Jul 29 21:12:30.118958 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.118862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn" event={"ID":"81839d31-2fd1-4365-8295-99e684928602","Type":"ContainerDied","Data":"1d0191f636c284e4cba5d221d1ddb30b7747be894f96e927fe80057426e32a1a"}
Jul 29 21:12:30.118958 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.118890    2581 scope.go:117] "RemoveContainer" containerID="d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"
Jul 29 21:12:30.127076 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.127057    2581 scope.go:117] "RemoveContainer" containerID="d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"
Jul 29 21:12:30.127388 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:12:30.127364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b\": container with ID starting with d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b not found: ID does not exist" containerID="d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"
Jul 29 21:12:30.127434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.127401    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b"} err="failed to get container status \"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b\": rpc error: code = NotFound desc = could not find container \"d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b\": container with ID starting with d5f3105be760966fad4e673610e99d4f5b4aac2c29adcaef3ca32a17118dd44b not found: ID does not exist"
Jul 29 21:12:30.160736 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgcs9\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.160866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.160866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.160990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.160990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.160990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.160987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.161132 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.161023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs\") pod \"81839d31-2fd1-4365-8295-99e684928602\" (UID: \"81839d31-2fd1-4365-8295-99e684928602\") "
Jul 29 21:12:30.161201 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.161157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:12:30.161292 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.161271    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81839d31-2fd1-4365-8295-99e684928602-istio-csr-ca-configmap\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.163320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163256    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:12:30.163464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163363    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts" (OuterVolumeSpecName: "cacerts") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:12:30.163522 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:12:30.163562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9" (OuterVolumeSpecName: "kube-api-access-mgcs9") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "kube-api-access-mgcs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:12:30.163639 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token" (OuterVolumeSpecName: "istio-token") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:12:30.163685 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.163634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs" (OuterVolumeSpecName: "local-certs") pod "81839d31-2fd1-4365-8295-99e684928602" (UID: "81839d31-2fd1-4365-8295-99e684928602"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:12:30.262328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262285    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-cacerts\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.262328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262320    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81839d31-2fd1-4365-8295-99e684928602-local-certs\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.262328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgcs9\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-kube-api-access-mgcs9\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.262328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262340    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81839d31-2fd1-4365-8295-99e684928602-istio-token\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.262593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262349    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-csr-dns-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.262593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.262357    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81839d31-2fd1-4365-8295-99e684928602-istio-kubeconfig\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:12:30.441554 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.441518    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:30.443887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:30.443856    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v48dn"]
Jul 29 21:12:31.185715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:31.185674    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81839d31-2fd1-4365-8295-99e684928602" path="/var/lib/kubelet/pods/81839d31-2fd1-4365-8295-99e684928602/volumes"
Jul 29 21:12:35.098423 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:35.098386    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mxbnj"
Jul 29 21:12:35.101712 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:12:35.101676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zzq88"
Jul 29 21:13:17.934832 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.934795    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:17.935273 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.935067    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81839d31-2fd1-4365-8295-99e684928602" containerName="discovery"
Jul 29 21:13:17.935273 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.935078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81839d31-2fd1-4365-8295-99e684928602" containerName="discovery"
Jul 29 21:13:17.935273 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.935121    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81839d31-2fd1-4365-8295-99e684928602" containerName="discovery"
Jul 29 21:13:17.936859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.936843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:17.939470 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.939445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 21:13:17.939949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.939925    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 21:13:17.940392 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.940378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-brbft\""
Jul 29 21:13:17.946833 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:17.946809    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:18.027560 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.027527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhch\" (UniqueName: \"kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch\") pod \"kuadrant-operator-catalog-8c265\" (UID: \"2da2c0c3-d151-4ac3-946d-742d2b7944e3\") " pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:18.128160 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.128128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhch\" (UniqueName: \"kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch\") pod \"kuadrant-operator-catalog-8c265\" (UID: \"2da2c0c3-d151-4ac3-946d-742d2b7944e3\") " pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:18.136778 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.136745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhch\" (UniqueName: \"kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch\") pod \"kuadrant-operator-catalog-8c265\" (UID: \"2da2c0c3-d151-4ac3-946d-742d2b7944e3\") " pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:18.247770 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.247656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:18.303109 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.303064    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:18.378387 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.378351    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:18.381533 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:18.381503    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da2c0c3_d151_4ac3_946d_742d2b7944e3.slice/crio-703213a82f3b0abc685a201f875c87d65dcb34556ab48539b9ab198999da547d WatchSource:0}: Error finding container 703213a82f3b0abc685a201f875c87d65dcb34556ab48539b9ab198999da547d: Status 404 returned error can't find the container with id 703213a82f3b0abc685a201f875c87d65dcb34556ab48539b9ab198999da547d
Jul 29 21:13:18.511825 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.511741    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fqfh4"]
Jul 29 21:13:18.514144 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.514127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:18.524828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.523716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fqfh4"]
Jul 29 21:13:18.632985 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.632947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j4z\" (UniqueName: \"kubernetes.io/projected/e89c8789-9fb4-4eac-92c5-cfe167a3a3b2-kube-api-access-g7j4z\") pod \"kuadrant-operator-catalog-fqfh4\" (UID: \"e89c8789-9fb4-4eac-92c5-cfe167a3a3b2\") " pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:18.733603 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.733562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j4z\" (UniqueName: \"kubernetes.io/projected/e89c8789-9fb4-4eac-92c5-cfe167a3a3b2-kube-api-access-g7j4z\") pod \"kuadrant-operator-catalog-fqfh4\" (UID: \"e89c8789-9fb4-4eac-92c5-cfe167a3a3b2\") " pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:18.745040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.745002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j4z\" (UniqueName: \"kubernetes.io/projected/e89c8789-9fb4-4eac-92c5-cfe167a3a3b2-kube-api-access-g7j4z\") pod \"kuadrant-operator-catalog-fqfh4\" (UID: \"e89c8789-9fb4-4eac-92c5-cfe167a3a3b2\") " pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:18.827630 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.827528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:18.965552 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:18.965526    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fqfh4"]
Jul 29 21:13:19.003537 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:19.003508    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89c8789_9fb4_4eac_92c5_cfe167a3a3b2.slice/crio-0718a848d6f32acc9b06d57f5d7c93323a9fc89c4f13bb3927390561838aa99f WatchSource:0}: Error finding container 0718a848d6f32acc9b06d57f5d7c93323a9fc89c4f13bb3927390561838aa99f: Status 404 returned error can't find the container with id 0718a848d6f32acc9b06d57f5d7c93323a9fc89c4f13bb3927390561838aa99f
Jul 29 21:13:19.324497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:19.323728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c265" event={"ID":"2da2c0c3-d151-4ac3-946d-742d2b7944e3","Type":"ContainerStarted","Data":"703213a82f3b0abc685a201f875c87d65dcb34556ab48539b9ab198999da547d"}
Jul 29 21:13:19.325935 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:19.325898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4" event={"ID":"e89c8789-9fb4-4eac-92c5-cfe167a3a3b2","Type":"ContainerStarted","Data":"0718a848d6f32acc9b06d57f5d7c93323a9fc89c4f13bb3927390561838aa99f"}
Jul 29 21:13:21.336468 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.336427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c265" event={"ID":"2da2c0c3-d151-4ac3-946d-742d2b7944e3","Type":"ContainerStarted","Data":"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"}
Jul 29 21:13:21.336989 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.336511    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8c265" podUID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" containerName="registry-server" containerID="cri-o://d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5" gracePeriod=2
Jul 29 21:13:21.337947 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.337913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4" event={"ID":"e89c8789-9fb4-4eac-92c5-cfe167a3a3b2","Type":"ContainerStarted","Data":"3ec726ad6862f3765914313d7e8fe4a10b0d0ebf43f91b0dfd674292c23f3be2"}
Jul 29 21:13:21.351715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.351658    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8c265" podStartSLOduration=2.196454647 podStartE2EDuration="4.351641879s" totalImagesPullingTime="2.155187232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:21.350864747 +0000 UTC m=+588.649342501" watchObservedRunningTime="2026-07-29 21:13:21.351641879 +0000 UTC m=+588.650119634"
Jul 29 21:13:21.365585 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.365538    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4" podStartSLOduration=1.828662049 podStartE2EDuration="3.365521926s" totalImagesPullingTime="1.536859877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:21.364147525 +0000 UTC m=+588.662625280" watchObservedRunningTime="2026-07-29 21:13:21.365521926 +0000 UTC m=+588.663999680"
Jul 29 21:13:21.577577 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.577551    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:21.656371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.656332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nhch\" (UniqueName: \"kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch\") pod \"2da2c0c3-d151-4ac3-946d-742d2b7944e3\" (UID: \"2da2c0c3-d151-4ac3-946d-742d2b7944e3\") "
Jul 29 21:13:21.658762 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.658732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch" (OuterVolumeSpecName: "kube-api-access-5nhch") pod "2da2c0c3-d151-4ac3-946d-742d2b7944e3" (UID: "2da2c0c3-d151-4ac3-946d-742d2b7944e3"). InnerVolumeSpecName "kube-api-access-5nhch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:13:21.756961 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:21.756923    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhch\" (UniqueName: \"kubernetes.io/projected/2da2c0c3-d151-4ac3-946d-742d2b7944e3-kube-api-access-5nhch\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:22.343300 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.343266    2581 generic.go:356] "Generic (PLEG): container finished" podID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" containerID="d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5" exitCode=0
Jul 29 21:13:22.343797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.343315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8c265"
Jul 29 21:13:22.343797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.343350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c265" event={"ID":"2da2c0c3-d151-4ac3-946d-742d2b7944e3","Type":"ContainerDied","Data":"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"}
Jul 29 21:13:22.343797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.343400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8c265" event={"ID":"2da2c0c3-d151-4ac3-946d-742d2b7944e3","Type":"ContainerDied","Data":"703213a82f3b0abc685a201f875c87d65dcb34556ab48539b9ab198999da547d"}
Jul 29 21:13:22.343797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.343420    2581 scope.go:117] "RemoveContainer" containerID="d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"
Jul 29 21:13:22.352804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.352787    2581 scope.go:117] "RemoveContainer" containerID="d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"
Jul 29 21:13:22.353069 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:13:22.353051    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5\": container with ID starting with d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5 not found: ID does not exist" containerID="d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"
Jul 29 21:13:22.353112 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.353078    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5"} err="failed to get container status \"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5\": rpc error: code = NotFound desc = could not find container \"d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5\": container with ID starting with d09c673c2ab87d154f9f35281ecf189ab2b76a74bac5b695bb2d1415be8850d5 not found: ID does not exist"
Jul 29 21:13:22.362758 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.362729    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:22.365515 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:22.365485    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8c265"]
Jul 29 21:13:23.185920 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:23.185885    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" path="/var/lib/kubelet/pods/2da2c0c3-d151-4ac3-946d-742d2b7944e3/volumes"
Jul 29 21:13:28.828491 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:28.828442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:28.829023 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:28.828522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:28.850410 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:28.850385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:29.393485 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:29.393457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fqfh4"
Jul 29 21:13:33.072542 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.072511    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:13:33.073804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.073783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:13:33.344820 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.344790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"]
Jul 29 21:13:33.345071 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.345059    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" containerName="registry-server"
Jul 29 21:13:33.345120 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.345073    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" containerName="registry-server"
Jul 29 21:13:33.345120 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.345112    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da2c0c3-d151-4ac3-946d-742d2b7944e3" containerName="registry-server"
Jul 29 21:13:33.349233 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.349214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.351490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.351466    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rgrr7\""
Jul 29 21:13:33.355463 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.355435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"]
Jul 29 21:13:33.447421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.447376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.447421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.447422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zb4n\" (UniqueName: \"kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.447674 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.447495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.548175 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.548121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.548339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.548188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zb4n\" (UniqueName: \"kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.548402 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.548360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.548596 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.548572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.548674 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.548656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.556751 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.556723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zb4n\" (UniqueName: \"kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.660081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.659987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:33.743788 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.743755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"]
Jul 29 21:13:33.753553 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.753514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.755218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.755187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"]
Jul 29 21:13:33.831191 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.831082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"]
Jul 29 21:13:33.833358 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:33.833331    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382fa42b_c64f_4c9d_a6b2_b2c7b929e319.slice/crio-6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81 WatchSource:0}: Error finding container 6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81: Status 404 returned error can't find the container with id 6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81
Jul 29 21:13:33.852669 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.852642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdd6\" (UniqueName: \"kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.852817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.852711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.852817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.852753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.954030 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.953986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.954030 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.954035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.954237 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.954078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdd6\" (UniqueName: \"kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.954398 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.954375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.954445 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.954405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:33.962371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:33.962345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdd6\" (UniqueName: \"kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:34.064821 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.064776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:34.145436 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.145406    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"]
Jul 29 21:13:34.150214 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.150032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.157767 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.157659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"]
Jul 29 21:13:34.203414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.203358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"]
Jul 29 21:13:34.205643 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:34.205619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c2979e_b10f_4c08_8da5_8cf8444f66ab.slice/crio-c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526 WatchSource:0}: Error finding container c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526: Status 404 returned error can't find the container with id c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526
Jul 29 21:13:34.256036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.255999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.256137 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.256058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csv54\" (UniqueName: \"kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.256189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.256145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.357186 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.357151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.357335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.357203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csv54\" (UniqueName: \"kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.357335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.357230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.357539 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.357524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.357611 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.357591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.365621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.365598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csv54\" (UniqueName: \"kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.394949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.394912    2581 generic.go:356] "Generic (PLEG): container finished" podID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerID="e88537e0fd142826fd392df2fd0532c26b3f96ccc7e806859f28afd53a13d22e" exitCode=0
Jul 29 21:13:34.395136 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.394996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerDied","Data":"e88537e0fd142826fd392df2fd0532c26b3f96ccc7e806859f28afd53a13d22e"}
Jul 29 21:13:34.395136 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.395054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerStarted","Data":"6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81"}
Jul 29 21:13:34.396679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.396655    2581 generic.go:356] "Generic (PLEG): container finished" podID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerID="1391f1c84ff2ecb6ee49070ccde5cb182dc85d6d4a598bc85256e31b2428a813" exitCode=0
Jul 29 21:13:34.396812 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.396690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerDied","Data":"1391f1c84ff2ecb6ee49070ccde5cb182dc85d6d4a598bc85256e31b2428a813"}
Jul 29 21:13:34.396812 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.396773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerStarted","Data":"c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526"}
Jul 29 21:13:34.462991 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.462893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:34.546956 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.546917    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"]
Jul 29 21:13:34.551360 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.551337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.557846 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.557819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"]
Jul 29 21:13:34.596401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.596365    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"]
Jul 29 21:13:34.599588 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:34.599556    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06b0603_f08e_4e5f_b8b3_b2685fe1de50.slice/crio-3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd WatchSource:0}: Error finding container 3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd: Status 404 returned error can't find the container with id 3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd
Jul 29 21:13:34.659310 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.659280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.659433 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.659379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknr4\" (UniqueName: \"kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.659433 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.659412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.760274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.760179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.760274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.760244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknr4\" (UniqueName: \"kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.760274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.760265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.760562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.760544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.760604 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.760576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.769284 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.769261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknr4\" (UniqueName: \"kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:34.878991 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:34.878954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:35.010745 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.010715    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"]
Jul 29 21:13:35.012731 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:35.012690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f9a784_1522_4a63_a361_96197dbce877.slice/crio-79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88 WatchSource:0}: Error finding container 79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88: Status 404 returned error can't find the container with id 79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88
Jul 29 21:13:35.404060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.404029    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7f9a784-1522-4a63-a361-96197dbce877" containerID="4f98a7166c8a56c3a9b854a8de2e3c2e9121305d1acc97db85f8cc27277e3c89" exitCode=0
Jul 29 21:13:35.404579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.404116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4" event={"ID":"f7f9a784-1522-4a63-a361-96197dbce877","Type":"ContainerDied","Data":"4f98a7166c8a56c3a9b854a8de2e3c2e9121305d1acc97db85f8cc27277e3c89"}
Jul 29 21:13:35.404579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.404178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4" event={"ID":"f7f9a784-1522-4a63-a361-96197dbce877","Type":"ContainerStarted","Data":"79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88"}
Jul 29 21:13:35.406776 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.406744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerStarted","Data":"080656bc330bd82f7f7d1d7045f9b4a2033d18ec1ef523ab7422ff8416f0d984"}
Jul 29 21:13:35.408628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.408604    2581 generic.go:356] "Generic (PLEG): container finished" podID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerID="753eae078113edd28c5e5a3021d54c75e5b4fc41c1bc14e8151fbca1faec0ae1" exitCode=0
Jul 29 21:13:35.408755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.408710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj" event={"ID":"c06b0603-f08e-4e5f-b8b3-b2685fe1de50","Type":"ContainerDied","Data":"753eae078113edd28c5e5a3021d54c75e5b4fc41c1bc14e8151fbca1faec0ae1"}
Jul 29 21:13:35.408824 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.408759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj" event={"ID":"c06b0603-f08e-4e5f-b8b3-b2685fe1de50","Type":"ContainerStarted","Data":"3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd"}
Jul 29 21:13:35.411345 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:35.411325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerStarted","Data":"d37fc9bad685e3522bb6fc68d6d208f4b901e55c551f3fade1890671e2ec4829"}
Jul 29 21:13:36.419181 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.419143    2581 generic.go:356] "Generic (PLEG): container finished" podID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerID="2e7eaae2bc292c423fc94b08a800bcca8ebd2dd6a047d6a0ce91d0ff0412c474" exitCode=0
Jul 29 21:13:36.419620 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.419197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj" event={"ID":"c06b0603-f08e-4e5f-b8b3-b2685fe1de50","Type":"ContainerDied","Data":"2e7eaae2bc292c423fc94b08a800bcca8ebd2dd6a047d6a0ce91d0ff0412c474"}
Jul 29 21:13:36.421298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.421278    2581 generic.go:356] "Generic (PLEG): container finished" podID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerID="d37fc9bad685e3522bb6fc68d6d208f4b901e55c551f3fade1890671e2ec4829" exitCode=0
Jul 29 21:13:36.421401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.421346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerDied","Data":"d37fc9bad685e3522bb6fc68d6d208f4b901e55c551f3fade1890671e2ec4829"}
Jul 29 21:13:36.423429 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.423408    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7f9a784-1522-4a63-a361-96197dbce877" containerID="4a022f7b970490ad2212f8c661cd0e0d910d0033051def79e879663db039f80f" exitCode=0
Jul 29 21:13:36.423554 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.423492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4" event={"ID":"f7f9a784-1522-4a63-a361-96197dbce877","Type":"ContainerDied","Data":"4a022f7b970490ad2212f8c661cd0e0d910d0033051def79e879663db039f80f"}
Jul 29 21:13:36.425651 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.425633    2581 generic.go:356] "Generic (PLEG): container finished" podID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerID="080656bc330bd82f7f7d1d7045f9b4a2033d18ec1ef523ab7422ff8416f0d984" exitCode=0
Jul 29 21:13:36.425760 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:36.425659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerDied","Data":"080656bc330bd82f7f7d1d7045f9b4a2033d18ec1ef523ab7422ff8416f0d984"}
Jul 29 21:13:37.432342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.432306    2581 generic.go:356] "Generic (PLEG): container finished" podID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerID="826575c2d91209e83c06d920bb4b3e2a6047c32538fbfc5575a392138d7c0090" exitCode=0
Jul 29 21:13:37.432780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.432397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerDied","Data":"826575c2d91209e83c06d920bb4b3e2a6047c32538fbfc5575a392138d7c0090"}
Jul 29 21:13:37.434413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.434392    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7f9a784-1522-4a63-a361-96197dbce877" containerID="51856efb7840b258faeb1aaa83952c86866370e7cd5fd3b5db1d088518ea2ac3" exitCode=0
Jul 29 21:13:37.434506 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.434471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4" event={"ID":"f7f9a784-1522-4a63-a361-96197dbce877","Type":"ContainerDied","Data":"51856efb7840b258faeb1aaa83952c86866370e7cd5fd3b5db1d088518ea2ac3"}
Jul 29 21:13:37.436536 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.436517    2581 generic.go:356] "Generic (PLEG): container finished" podID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerID="60dc86611d14e7c647cc58ed5b396e3ddc389e38bc73ffa46c33024e61482eb6" exitCode=0
Jul 29 21:13:37.436637 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.436588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerDied","Data":"60dc86611d14e7c647cc58ed5b396e3ddc389e38bc73ffa46c33024e61482eb6"}
Jul 29 21:13:37.438440 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.438417    2581 generic.go:356] "Generic (PLEG): container finished" podID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerID="d2a8ef685be52bf4d3d027315b859c39a4884ec74dfce5dbfa8f65187e53d3d6" exitCode=0
Jul 29 21:13:37.438510 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:37.438451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj" event={"ID":"c06b0603-f08e-4e5f-b8b3-b2685fe1de50","Type":"ContainerDied","Data":"d2a8ef685be52bf4d3d027315b859c39a4884ec74dfce5dbfa8f65187e53d3d6"}
Jul 29 21:13:38.610033 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.609998    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:38.633892 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.633852    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:38.637897 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.637873    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:38.643873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.643848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:38.696119 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtdd6\" (UniqueName: \"kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6\") pod \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") "
Jul 29 21:13:38.696352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util\") pod \"f7f9a784-1522-4a63-a361-96197dbce877\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") "
Jul 29 21:13:38.696352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle\") pod \"f7f9a784-1522-4a63-a361-96197dbce877\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") "
Jul 29 21:13:38.696352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle\") pod \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") "
Jul 29 21:13:38.696352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kknr4\" (UniqueName: \"kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4\") pod \"f7f9a784-1522-4a63-a361-96197dbce877\" (UID: \"f7f9a784-1522-4a63-a361-96197dbce877\") "
Jul 29 21:13:38.696352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zb4n\" (UniqueName: \"kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n\") pod \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") "
Jul 29 21:13:38.696664 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.696374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util\") pod \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\" (UID: \"382fa42b-c64f-4c9d-a6b2-b2c7b929e319\") "
Jul 29 21:13:38.697223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.697155    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle" (OuterVolumeSpecName: "bundle") pod "382fa42b-c64f-4c9d-a6b2-b2c7b929e319" (UID: "382fa42b-c64f-4c9d-a6b2-b2c7b929e319"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.698334 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.698301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle" (OuterVolumeSpecName: "bundle") pod "f7f9a784-1522-4a63-a361-96197dbce877" (UID: "f7f9a784-1522-4a63-a361-96197dbce877"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.698871 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.698841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6" (OuterVolumeSpecName: "kube-api-access-vtdd6") pod "60c2979e-b10f-4c08-8da5-8cf8444f66ab" (UID: "60c2979e-b10f-4c08-8da5-8cf8444f66ab"). InnerVolumeSpecName "kube-api-access-vtdd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:13:38.699657 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.699627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4" (OuterVolumeSpecName: "kube-api-access-kknr4") pod "f7f9a784-1522-4a63-a361-96197dbce877" (UID: "f7f9a784-1522-4a63-a361-96197dbce877"). InnerVolumeSpecName "kube-api-access-kknr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:13:38.699777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.699690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n" (OuterVolumeSpecName: "kube-api-access-7zb4n") pod "382fa42b-c64f-4c9d-a6b2-b2c7b929e319" (UID: "382fa42b-c64f-4c9d-a6b2-b2c7b929e319"). InnerVolumeSpecName "kube-api-access-7zb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:13:38.703130 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.703105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util" (OuterVolumeSpecName: "util") pod "382fa42b-c64f-4c9d-a6b2-b2c7b929e319" (UID: "382fa42b-c64f-4c9d-a6b2-b2c7b929e319"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.704147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.704123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util" (OuterVolumeSpecName: "util") pod "f7f9a784-1522-4a63-a361-96197dbce877" (UID: "f7f9a784-1522-4a63-a361-96197dbce877"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.797302 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csv54\" (UniqueName: \"kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54\") pod \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") "
Jul 29 21:13:38.797478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle\") pod \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") "
Jul 29 21:13:38.797478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle\") pod \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") "
Jul 29 21:13:38.797478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util\") pod \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\" (UID: \"c06b0603-f08e-4e5f-b8b3-b2685fe1de50\") "
Jul 29 21:13:38.797478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util\") pod \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\" (UID: \"60c2979e-b10f-4c08-8da5-8cf8444f66ab\") "
Jul 29 21:13:38.797644 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797628    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknr4\" (UniqueName: \"kubernetes.io/projected/f7f9a784-1522-4a63-a361-96197dbce877-kube-api-access-kknr4\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797650    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zb4n\" (UniqueName: \"kubernetes.io/projected/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-kube-api-access-7zb4n\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797667    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797811 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797681    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdd6\" (UniqueName: \"kubernetes.io/projected/60c2979e-b10f-4c08-8da5-8cf8444f66ab-kube-api-access-vtdd6\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797811 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797726    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797811 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797741    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7f9a784-1522-4a63-a361-96197dbce877-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797811 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797753    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/382fa42b-c64f-4c9d-a6b2-b2c7b929e319-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.797995 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.797961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle" (OuterVolumeSpecName: "bundle") pod "c06b0603-f08e-4e5f-b8b3-b2685fe1de50" (UID: "c06b0603-f08e-4e5f-b8b3-b2685fe1de50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.798214 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.798145    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle" (OuterVolumeSpecName: "bundle") pod "60c2979e-b10f-4c08-8da5-8cf8444f66ab" (UID: "60c2979e-b10f-4c08-8da5-8cf8444f66ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.799612 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.799578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54" (OuterVolumeSpecName: "kube-api-access-csv54") pod "c06b0603-f08e-4e5f-b8b3-b2685fe1de50" (UID: "c06b0603-f08e-4e5f-b8b3-b2685fe1de50"). InnerVolumeSpecName "kube-api-access-csv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:13:38.803217 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.803178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util" (OuterVolumeSpecName: "util") pod "c06b0603-f08e-4e5f-b8b3-b2685fe1de50" (UID: "c06b0603-f08e-4e5f-b8b3-b2685fe1de50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.803300 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.803189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util" (OuterVolumeSpecName: "util") pod "60c2979e-b10f-4c08-8da5-8cf8444f66ab" (UID: "60c2979e-b10f-4c08-8da5-8cf8444f66ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:13:38.898396 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.898364    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csv54\" (UniqueName: \"kubernetes.io/projected/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-kube-api-access-csv54\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.898396 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.898395    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.898396 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.898405    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-bundle\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.898641 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.898413    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c06b0603-f08e-4e5f-b8b3-b2685fe1de50-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:38.898641 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:38.898421    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c2979e-b10f-4c08-8da5-8cf8444f66ab-util\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:13:39.453178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.453087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4"
Jul 29 21:13:39.453178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.453095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4" event={"ID":"f7f9a784-1522-4a63-a361-96197dbce877","Type":"ContainerDied","Data":"79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88"}
Jul 29 21:13:39.453178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.453149    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d1476a2f1d54a9c03f73f2308c14e664d06fd708115c4b745a45774c500e88"
Jul 29 21:13:39.455288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.455264    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x"
Jul 29 21:13:39.455558 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.455262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x" event={"ID":"382fa42b-c64f-4c9d-a6b2-b2c7b929e319","Type":"ContainerDied","Data":"6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81"}
Jul 29 21:13:39.455663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.455583    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6690d38f2158b995a970d064c726d6077bd1a00e631ce5dcee1e8d02d21c7d81"
Jul 29 21:13:39.457246 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.457222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj" event={"ID":"c06b0603-f08e-4e5f-b8b3-b2685fe1de50","Type":"ContainerDied","Data":"3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd"}
Jul 29 21:13:39.457350 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.457278    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6428afad3713fe2a57b33939f5dfee50fb30cead3e8d652f8c2140141292bd"
Jul 29 21:13:39.457350 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.457239    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj"
Jul 29 21:13:39.459884 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.459860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8" event={"ID":"60c2979e-b10f-4c08-8da5-8cf8444f66ab","Type":"ContainerDied","Data":"c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526"}
Jul 29 21:13:39.459884 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.459881    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8"
Jul 29 21:13:39.460048 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:39.459897    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0ac85751d2650671606a01a44fee8e755af34a9f72d74efa49d92599eec2526"
Jul 29 21:13:45.382243 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382203    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"]
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382489    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382509    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382520    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382526    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382537    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382545    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382568    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382573    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382579    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382595    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="util"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382600    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382604    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382612    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382618    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382643    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="pull"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c2979e-b10f-4c08-8da5-8cf8444f66ab" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="382fa42b-c64f-4c9d-a6b2-b2c7b929e319" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c06b0603-f08e-4e5f-b8b3-b2685fe1de50" containerName="extract"
Jul 29 21:13:45.382755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.382740    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f9a784-1522-4a63-a361-96197dbce877" containerName="extract"
Jul 29 21:13:45.386951 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.386933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.390786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.390762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4hrql\""
Jul 29 21:13:45.399030 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.398999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"]
Jul 29 21:13:45.549565 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.549523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpc4\" (UniqueName: \"kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.549784 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.549612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.650926 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.650822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpc4\" (UniqueName: \"kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.650926 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.650909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.651261 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.651244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.666758 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.666722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpc4\" (UniqueName: \"kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.696779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.696740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:45.837064 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:45.836983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"]
Jul 29 21:13:45.839440 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:45.839415    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30505586_f169_478d_828c_8e89c17011bd.slice/crio-edacbbd276b440710270802560f6381e784863ed220fbc8932a7fcb0b5366360 WatchSource:0}: Error finding container edacbbd276b440710270802560f6381e784863ed220fbc8932a7fcb0b5366360: Status 404 returned error can't find the container with id edacbbd276b440710270802560f6381e784863ed220fbc8932a7fcb0b5366360
Jul 29 21:13:46.493680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:46.493649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4" event={"ID":"30505586-f169-478d-828c-8e89c17011bd","Type":"ContainerStarted","Data":"edacbbd276b440710270802560f6381e784863ed220fbc8932a7fcb0b5366360"}
Jul 29 21:13:47.885529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:47.885485    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"]
Jul 29 21:13:47.889035 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:47.889008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:47.891518 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:47.891488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7kcv2\""
Jul 29 21:13:47.900427 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:47.900399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"]
Jul 29 21:13:47.969077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:47.969032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9dq\" (UniqueName: \"kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq\") pod \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" (UID: \"ce8ca9e0-86ee-4891-b311-a4d9034e3955\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:48.069594 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:48.069553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9dq\" (UniqueName: \"kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq\") pod \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" (UID: \"ce8ca9e0-86ee-4891-b311-a4d9034e3955\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:48.079921 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:48.079886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9dq\" (UniqueName: \"kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq\") pod \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" (UID: \"ce8ca9e0-86ee-4891-b311-a4d9034e3955\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:48.203384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:48.203299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:48.346487 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:48.346452    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"]
Jul 29 21:13:48.349376 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:48.349343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8ca9e0_86ee_4891_b311_a4d9034e3955.slice/crio-f4fc4fb85fff8e6e4307f7484a2828bba30294113fd7c4ec57007a10c433faf0 WatchSource:0}: Error finding container f4fc4fb85fff8e6e4307f7484a2828bba30294113fd7c4ec57007a10c433faf0: Status 404 returned error can't find the container with id f4fc4fb85fff8e6e4307f7484a2828bba30294113fd7c4ec57007a10c433faf0
Jul 29 21:13:48.504403 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:48.504308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9" event={"ID":"ce8ca9e0-86ee-4891-b311-a4d9034e3955","Type":"ContainerStarted","Data":"f4fc4fb85fff8e6e4307f7484a2828bba30294113fd7c4ec57007a10c433faf0"}
Jul 29 21:13:50.764914 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.764882    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d94c59877-9vf9k"]
Jul 29 21:13:50.767985 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.767961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.771634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.771606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 21:13:50.771817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.771605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 21:13:50.771817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.771667    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 21:13:50.771817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.771671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 21:13:50.771817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.771609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-64d8z\""
Jul 29 21:13:50.772135 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.772121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 21:13:50.772181 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.772148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 21:13:50.772222 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.772202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 21:13:50.777890 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.777851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 21:13:50.780820 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.780796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d94c59877-9vf9k"]
Jul 29 21:13:50.893067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-oauth-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-oauth-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvnd\" (UniqueName: \"kubernetes.io/projected/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-kube-api-access-znvnd\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-trusted-ca-bundle\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.893444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.893335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-service-ca\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.994555 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-oauth-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.994781 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.994781 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-oauth-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.994781 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvnd\" (UniqueName: \"kubernetes.io/projected/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-kube-api-access-znvnd\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.994781 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.995007 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-trusted-ca-bundle\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.995007 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.994832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-service-ca\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.995816 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.995788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.995971 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.995944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-oauth-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.996089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.996060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-trusted-ca-bundle\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.996477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.996455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-service-ca\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.997658 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.997615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-oauth-config\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:50.997790 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:50.997772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-console-serving-cert\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:51.003399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.003377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvnd\" (UniqueName: \"kubernetes.io/projected/4c2cc28a-8269-4fd7-9a46-004bfb3a4835-kube-api-access-znvnd\") pod \"console-d94c59877-9vf9k\" (UID: \"4c2cc28a-8269-4fd7-9a46-004bfb3a4835\") " pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:51.080475 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.080394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:13:51.276518 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.276488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d94c59877-9vf9k"]
Jul 29 21:13:51.279153 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:51.279123    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2cc28a_8269_4fd7_9a46_004bfb3a4835.slice/crio-f77e2b940b6e1e193c46967fb4a70a873b2b2897b9220b60721c2aae995a0207 WatchSource:0}: Error finding container f77e2b940b6e1e193c46967fb4a70a873b2b2897b9220b60721c2aae995a0207: Status 404 returned error can't find the container with id f77e2b940b6e1e193c46967fb4a70a873b2b2897b9220b60721c2aae995a0207
Jul 29 21:13:51.382742 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.382684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"]
Jul 29 21:13:51.386406 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.386376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:51.388768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.388746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g4pn2\""
Jul 29 21:13:51.388891 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.388748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 21:13:51.396715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.396670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"]
Jul 29 21:13:51.499628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.499587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v79\" (UniqueName: \"kubernetes.io/projected/70072b7d-14d8-4e9b-949a-ce93034a8c7a-kube-api-access-85v79\") pod \"dns-operator-controller-manager-864dd78c4c-8njwz\" (UID: \"70072b7d-14d8-4e9b-949a-ce93034a8c7a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:51.521426 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.521391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9" event={"ID":"ce8ca9e0-86ee-4891-b311-a4d9034e3955","Type":"ContainerStarted","Data":"67379a8919f1d76c76af2c637a912200195ee007dc62cc7e39aafcb7d8704996"}
Jul 29 21:13:51.522137 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.522028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:13:51.528029 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.528001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94c59877-9vf9k" event={"ID":"4c2cc28a-8269-4fd7-9a46-004bfb3a4835","Type":"ContainerStarted","Data":"6e6b84a343fb6deba83f6a41a4e8a492587a35f8f9467b0dd83409911ab21291"}
Jul 29 21:13:51.528174 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.528053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d94c59877-9vf9k" event={"ID":"4c2cc28a-8269-4fd7-9a46-004bfb3a4835","Type":"ContainerStarted","Data":"f77e2b940b6e1e193c46967fb4a70a873b2b2897b9220b60721c2aae995a0207"}
Jul 29 21:13:51.529687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.529661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4" event={"ID":"30505586-f169-478d-828c-8e89c17011bd","Type":"ContainerStarted","Data":"3061fc239bd8c351350ddbd2da3aefb1c89aabf4b1d31dd53bd3266b00762255"}
Jul 29 21:13:51.529806 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.529749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:13:51.541762 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.541711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9" podStartSLOduration=1.7580564810000001 podStartE2EDuration="4.54168124s" totalImagesPullingTime="2.783624759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:51.540147493 +0000 UTC m=+618.838625248" watchObservedRunningTime="2026-07-29 21:13:51.54168124 +0000 UTC m=+618.840158996"
Jul 29 21:13:51.563530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.563464    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d94c59877-9vf9k" podStartSLOduration=1.563444992 podStartE2EDuration="1.563444992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:51.562435618 +0000 UTC m=+618.860913374" watchObservedRunningTime="2026-07-29 21:13:51.563444992 +0000 UTC m=+618.861922741"
Jul 29 21:13:51.582353 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.582176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4" podStartSLOduration=1.283281073 podStartE2EDuration="6.582154521s" totalImagesPullingTime="5.298873448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:51.581484601 +0000 UTC m=+618.879962426" watchObservedRunningTime="2026-07-29 21:13:51.582154521 +0000 UTC m=+618.880632276"
Jul 29 21:13:51.600628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.600589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v79\" (UniqueName: \"kubernetes.io/projected/70072b7d-14d8-4e9b-949a-ce93034a8c7a-kube-api-access-85v79\") pod \"dns-operator-controller-manager-864dd78c4c-8njwz\" (UID: \"70072b7d-14d8-4e9b-949a-ce93034a8c7a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:51.612640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.612607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v79\" (UniqueName: \"kubernetes.io/projected/70072b7d-14d8-4e9b-949a-ce93034a8c7a-kube-api-access-85v79\") pod \"dns-operator-controller-manager-864dd78c4c-8njwz\" (UID: \"70072b7d-14d8-4e9b-949a-ce93034a8c7a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:51.699295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.699179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:51.887903 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:51.887791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"]
Jul 29 21:13:51.890125 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:51.890090    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70072b7d_14d8_4e9b_949a_ce93034a8c7a.slice/crio-1b5e7039e800f3486fc4b8ca66cc03fa607dfc70fc60a155366473f0ebbc4757 WatchSource:0}: Error finding container 1b5e7039e800f3486fc4b8ca66cc03fa607dfc70fc60a155366473f0ebbc4757: Status 404 returned error can't find the container with id 1b5e7039e800f3486fc4b8ca66cc03fa607dfc70fc60a155366473f0ebbc4757
Jul 29 21:13:52.536006 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:52.535968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz" event={"ID":"70072b7d-14d8-4e9b-949a-ce93034a8c7a","Type":"ContainerStarted","Data":"1b5e7039e800f3486fc4b8ca66cc03fa607dfc70fc60a155366473f0ebbc4757"}
Jul 29 21:13:54.166065 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.166025    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"]
Jul 29 21:13:54.169348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.169321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.171436 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.171413    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 21:13:54.171566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.171445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 21:13:54.171566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.171454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rgrr7\""
Jul 29 21:13:54.178096 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.178072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"]
Jul 29 21:13:54.326904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.326817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c294ae37-68cc-4277-a641-086991f8f09f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.326904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.326868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hjm\" (UniqueName: \"kubernetes.io/projected/c294ae37-68cc-4277-a641-086991f8f09f-kube-api-access-g7hjm\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.326904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.326889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c294ae37-68cc-4277-a641-086991f8f09f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.428218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.428180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c294ae37-68cc-4277-a641-086991f8f09f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.428218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.428231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hjm\" (UniqueName: \"kubernetes.io/projected/c294ae37-68cc-4277-a641-086991f8f09f-kube-api-access-g7hjm\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.428444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.428259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c294ae37-68cc-4277-a641-086991f8f09f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.429022 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.429003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c294ae37-68cc-4277-a641-086991f8f09f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.430905 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.430889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c294ae37-68cc-4277-a641-086991f8f09f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.436628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.436586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hjm\" (UniqueName: \"kubernetes.io/projected/c294ae37-68cc-4277-a641-086991f8f09f-kube-api-access-g7hjm\") pod \"kuadrant-console-plugin-66b5774985-zfkl8\" (UID: \"c294ae37-68cc-4277-a641-086991f8f09f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.478768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.478729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"
Jul 29 21:13:54.550047 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.550011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz" event={"ID":"70072b7d-14d8-4e9b-949a-ce93034a8c7a","Type":"ContainerStarted","Data":"d17e2014caa0edffea0fde9edb7239e73c42b6252607e507d3d963e6bad49b47"}
Jul 29 21:13:54.550211 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.550187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:13:54.568147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.568093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz" podStartSLOduration=1.467397769 podStartE2EDuration="3.568072378s" totalImagesPullingTime="2.100674609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:54.565992913 +0000 UTC m=+621.864470670" watchObservedRunningTime="2026-07-29 21:13:54.568072378 +0000 UTC m=+621.866550133"
Jul 29 21:13:54.621007 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:54.620970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8"]
Jul 29 21:13:54.623884 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:54.623847    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc294ae37_68cc_4277_a641_086991f8f09f.slice/crio-6dd016879b6e9f896f02a3e9806538fb809ff93c1d6620610ef1854ef3efa388 WatchSource:0}: Error finding container 6dd016879b6e9f896f02a3e9806538fb809ff93c1d6620610ef1854ef3efa388: Status 404 returned error can't find the container with id 6dd016879b6e9f896f02a3e9806538fb809ff93c1d6620610ef1854ef3efa388
Jul 29 21:13:55.557910 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.557824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8" event={"ID":"c294ae37-68cc-4277-a641-086991f8f09f","Type":"ContainerStarted","Data":"6dd016879b6e9f896f02a3e9806538fb809ff93c1d6620610ef1854ef3efa388"}
Jul 29 21:13:55.783991 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.783951    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"]
Jul 29 21:13:55.786899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.786878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:13:55.789998 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.789974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tktd8\""
Jul 29 21:13:55.806347 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.806318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"]
Jul 29 21:13:55.943241 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:55.943193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqn2k\" (UniqueName: \"kubernetes.io/projected/ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25-kube-api-access-tqn2k\") pod \"authorino-operator-75cc8d5cb7-kxvdz\" (UID: \"ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:13:56.044329 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.044284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqn2k\" (UniqueName: \"kubernetes.io/projected/ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25-kube-api-access-tqn2k\") pod \"authorino-operator-75cc8d5cb7-kxvdz\" (UID: \"ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:13:56.070040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.070004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqn2k\" (UniqueName: \"kubernetes.io/projected/ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25-kube-api-access-tqn2k\") pod \"authorino-operator-75cc8d5cb7-kxvdz\" (UID: \"ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:13:56.100876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.100743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:13:56.275466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.275432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"]
Jul 29 21:13:56.282271 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:13:56.281764    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef69bd10_a7ee_49c0_9478_c3ccb8e9ff25.slice/crio-30c497b81873c12d31a283a4250413aff1879bb506a777a217decb4986b639bb WatchSource:0}: Error finding container 30c497b81873c12d31a283a4250413aff1879bb506a777a217decb4986b639bb: Status 404 returned error can't find the container with id 30c497b81873c12d31a283a4250413aff1879bb506a777a217decb4986b639bb
Jul 29 21:13:56.286551 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.286473    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:13:56.566466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:13:56.566371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz" event={"ID":"ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25","Type":"ContainerStarted","Data":"30c497b81873c12d31a283a4250413aff1879bb506a777a217decb4986b639bb"}
Jul 29 21:14:00.592552 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:00.592519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz" event={"ID":"ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25","Type":"ContainerStarted","Data":"1624969a5525c6730ab6ef4119d7ce87ca3590f24a14a54a9c85ec1097083ec2"}
Jul 29 21:14:00.593012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:00.592670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:14:00.609846 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:00.609790    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz" podStartSLOduration=1.5264840149999999 podStartE2EDuration="5.609772634s" totalImagesPullingTime="4.083288619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:00.607605184 +0000 UTC m=+627.906082940" watchObservedRunningTime="2026-07-29 21:14:00.609772634 +0000 UTC m=+627.908250400"
Jul 29 21:14:01.080603 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:01.080564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:14:01.080815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:01.080632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:14:01.087289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:01.087262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:14:01.602981 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:01.602944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d94c59877-9vf9k"
Jul 29 21:14:02.538646 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:02.538610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:02.539020 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:02.538995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:04.098189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.098149    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"]
Jul 29 21:14:04.115903 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.115860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"]
Jul 29 21:14:04.116085 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.115967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.216710 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.216666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4k8\" (UniqueName: \"kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.216889 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.216754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.317327 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.317279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.317513 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.317399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4k8\" (UniqueName: \"kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.317769 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.317749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.325363 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.325339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4k8\" (UniqueName: \"kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8\") pod \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.400974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.400893    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"]
Jul 29 21:14:04.401213 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.401198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:04.412656 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.410867    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"]
Jul 29 21:14:04.423659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.423618    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:04.427087 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.427065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.443440 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.443345    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"]
Jul 29 21:14:04.443739 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.443685    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4" podUID="30505586-f169-478d-828c-8e89c17011bd" containerName="manager" containerID="cri-o://3061fc239bd8c351350ddbd2da3aefb1c89aabf4b1d31dd53bd3266b00762255" gracePeriod=2
Jul 29 21:14:04.446202 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.446166    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:04.459539 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.459507    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"]
Jul 29 21:14:04.465378 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.465341    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:04.470483 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.470445    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"]
Jul 29 21:14:04.470808 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.470746    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" containerName="manager" containerID="cri-o://67379a8919f1d76c76af2c637a912200195ee007dc62cc7e39aafcb7d8704996" gracePeriod=2
Jul 29 21:14:04.473233 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.473185    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:04.474175 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.473862    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:04.474357 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.474338    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30505586-f169-478d-828c-8e89c17011bd" containerName="manager"
Jul 29 21:14:04.474423 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.474361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="30505586-f169-478d-828c-8e89c17011bd" containerName="manager"
Jul 29 21:14:04.474490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.474471    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="30505586-f169-478d-828c-8e89c17011bd" containerName="manager"
Jul 29 21:14:04.475512 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.475477    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:04.478004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.477980    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"]
Jul 29 21:14:04.478101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.478087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.480404 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.480273    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:04.482349 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.482127    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:04.488268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.488186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:04.494183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.494156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"]
Jul 29 21:14:04.494535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.494517    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" containerName="manager"
Jul 29 21:14:04.494535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.494536    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" containerName="manager"
Jul 29 21:14:04.494690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.494619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" containerName="manager"
Jul 29 21:14:04.497896 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.497873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:04.508337 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.508294    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:04.514405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.514376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"]
Jul 29 21:14:04.518950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.518913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mntv\" (UniqueName: \"kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.519121 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.518967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8964g\" (UniqueName: \"kubernetes.io/projected/30b70b50-e020-47cc-b1e0-c00cc9f167e9-kube-api-access-8964g\") pod \"limitador-operator-controller-manager-859d7b78bd-m5szc\" (UID: \"30b70b50-e020-47cc-b1e0-c00cc9f167e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:04.519121 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.519049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.519121 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.519109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.519255 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.519181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7ld\" (UniqueName: \"kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.532189 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:04.532147    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:04.619792 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.619757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mntv\" (UniqueName: \"kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.619792 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.619799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8964g\" (UniqueName: \"kubernetes.io/projected/30b70b50-e020-47cc-b1e0-c00cc9f167e9-kube-api-access-8964g\") pod \"limitador-operator-controller-manager-859d7b78bd-m5szc\" (UID: \"30b70b50-e020-47cc-b1e0-c00cc9f167e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:04.620066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.619835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.620066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.619865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.620066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.619905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7ld\" (UniqueName: \"kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.620323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.620297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.620408 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.620382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.628975 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.628946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mntv\" (UniqueName: \"kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bpr46\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.629125 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.628946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8964g\" (UniqueName: \"kubernetes.io/projected/30b70b50-e020-47cc-b1e0-c00cc9f167e9-kube-api-access-8964g\") pod \"limitador-operator-controller-manager-859d7b78bd-m5szc\" (UID: \"30b70b50-e020-47cc-b1e0-c00cc9f167e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:04.629125 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.629040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7ld\" (UniqueName: \"kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld\") pod \"kuadrant-operator-controller-manager-55467d9f7-6g5nz\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.740402 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.740307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:04.853546 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.853502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:04.861554 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:04.861519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:05.560288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:05.560254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8njwz"
Jul 29 21:14:11.601288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:11.600949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxvdz"
Jul 29 21:14:16.685240 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.685178    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" containerID="67379a8919f1d76c76af2c637a912200195ee007dc62cc7e39aafcb7d8704996" exitCode=0
Jul 29 21:14:16.687538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.687513    2581 generic.go:356] "Generic (PLEG): container finished" podID="30505586-f169-478d-828c-8e89c17011bd" containerID="3061fc239bd8c351350ddbd2da3aefb1c89aabf4b1d31dd53bd3266b00762255" exitCode=0
Jul 29 21:14:16.837142 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.837049    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:16.840267 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:16.840227    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:16.849593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.849567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:16.852257 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:16.852126    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:16.854650 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:16.854608    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:16.872861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.871562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:16.875540 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:16.875511    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7687cf70_deda_4ce1_8dd4_71805e12e3fa.slice/crio-1d5b5fb48a5b336be39a801556d66f54661df7488e1c8505cbbd38318231d33f WatchSource:0}: Error finding container 1d5b5fb48a5b336be39a801556d66f54661df7488e1c8505cbbd38318231d33f: Status 404 returned error can't find the container with id 1d5b5fb48a5b336be39a801556d66f54661df7488e1c8505cbbd38318231d33f
Jul 29 21:14:16.891081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.891049    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:16.893896 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:16.893815    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70f1416_ab1e_415d_8b30_beff882874aa.slice/crio-546c41f52b281483f54ffe597604aa93ef595585dcffafc51f09467c29d22c1d WatchSource:0}: Error finding container 546c41f52b281483f54ffe597604aa93ef595585dcffafc51f09467c29d22c1d: Status 404 returned error can't find the container with id 546c41f52b281483f54ffe597604aa93ef595585dcffafc51f09467c29d22c1d
Jul 29 21:14:16.911245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.911210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"]
Jul 29 21:14:16.914103 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:16.914075    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b70b50_e020_47cc_b1e0_c00cc9f167e9.slice/crio-620d5e337dccda75e4d7458cd81822f2d555acdaaa71db959101cc6e880abbe7 WatchSource:0}: Error finding container 620d5e337dccda75e4d7458cd81822f2d555acdaaa71db959101cc6e880abbe7: Status 404 returned error can't find the container with id 620d5e337dccda75e4d7458cd81822f2d555acdaaa71db959101cc6e880abbe7
Jul 29 21:14:16.936856 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.936835    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzpc4\" (UniqueName: \"kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4\") pod \"30505586-f169-478d-828c-8e89c17011bd\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") "
Jul 29 21:14:16.936990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.936872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume\") pod \"30505586-f169-478d-828c-8e89c17011bd\" (UID: \"30505586-f169-478d-828c-8e89c17011bd\") "
Jul 29 21:14:16.936990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.936934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b9dq\" (UniqueName: \"kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq\") pod \"ce8ca9e0-86ee-4891-b311-a4d9034e3955\" (UID: \"ce8ca9e0-86ee-4891-b311-a4d9034e3955\") "
Jul 29 21:14:16.937543 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.937508    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30505586-f169-478d-828c-8e89c17011bd" (UID: "30505586-f169-478d-828c-8e89c17011bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:16.939507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.939464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4" (OuterVolumeSpecName: "kube-api-access-wzpc4") pod "30505586-f169-478d-828c-8e89c17011bd" (UID: "30505586-f169-478d-828c-8e89c17011bd"). InnerVolumeSpecName "kube-api-access-wzpc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:16.940020 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:16.939997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq" (OuterVolumeSpecName: "kube-api-access-2b9dq") pod "ce8ca9e0-86ee-4891-b311-a4d9034e3955" (UID: "ce8ca9e0-86ee-4891-b311-a4d9034e3955"). InnerVolumeSpecName "kube-api-access-2b9dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:17.038405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.038361    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpc4\" (UniqueName: \"kubernetes.io/projected/30505586-f169-478d-828c-8e89c17011bd-kube-api-access-wzpc4\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:17.038405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.038395    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30505586-f169-478d-828c-8e89c17011bd-extensions-socket-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:17.038405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.038411    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9dq\" (UniqueName: \"kubernetes.io/projected/ce8ca9e0-86ee-4891-b311-a4d9034e3955-kube-api-access-2b9dq\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:17.186159 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.186080    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30505586-f169-478d-828c-8e89c17011bd" path="/var/lib/kubelet/pods/30505586-f169-478d-828c-8e89c17011bd/volumes"
Jul 29 21:14:17.186496 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.186480    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" path="/var/lib/kubelet/pods/ce8ca9e0-86ee-4891-b311-a4d9034e3955/volumes"
Jul 29 21:14:17.694265 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.694229    2581 scope.go:117] "RemoveContainer" containerID="67379a8919f1d76c76af2c637a912200195ee007dc62cc7e39aafcb7d8704996"
Jul 29 21:14:17.694739 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.694232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:17.696223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.696190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" event={"ID":"a70f1416-ab1e-415d-8b30-beff882874aa","Type":"ContainerStarted","Data":"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"}
Jul 29 21:14:17.696344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.696258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" event={"ID":"a70f1416-ab1e-415d-8b30-beff882874aa","Type":"ContainerStarted","Data":"546c41f52b281483f54ffe597604aa93ef595585dcffafc51f09467c29d22c1d"}
Jul 29 21:14:17.696407 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.696384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:17.696934 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:17.696899    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:17.698221 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.698196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" event={"ID":"7687cf70-deda-4ce1-8dd4-71805e12e3fa","Type":"ContainerStarted","Data":"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"}
Jul 29 21:14:17.698336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.698245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:17.698336 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.698261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" event={"ID":"7687cf70-deda-4ce1-8dd4-71805e12e3fa","Type":"ContainerStarted","Data":"1d5b5fb48a5b336be39a801556d66f54661df7488e1c8505cbbd38318231d33f"}
Jul 29 21:14:17.698834 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:17.698809    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:17.699809 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.699784    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:17.701552 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.701502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8" event={"ID":"c294ae37-68cc-4277-a641-086991f8f09f","Type":"ContainerStarted","Data":"7c32c7e244b7a1cf77281efa0b902d4dad02dcd43ba1ad0a055cbd5029c6a079"}
Jul 29 21:14:17.703279 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.703258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc" event={"ID":"30b70b50-e020-47cc-b1e0-c00cc9f167e9","Type":"ContainerStarted","Data":"814fc08492c90dc28c2db981d251327d37dde4470eda9c0b4172c13bcab75196"}
Jul 29 21:14:17.703377 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.703297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc" event={"ID":"30b70b50-e020-47cc-b1e0-c00cc9f167e9","Type":"ContainerStarted","Data":"620d5e337dccda75e4d7458cd81822f2d555acdaaa71db959101cc6e880abbe7"}
Jul 29 21:14:17.703377 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.703356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:17.705039 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.705023    2581 scope.go:117] "RemoveContainer" containerID="3061fc239bd8c351350ddbd2da3aefb1c89aabf4b1d31dd53bd3266b00762255"
Jul 29 21:14:17.717682 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.717632    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" podStartSLOduration=13.717615688 podStartE2EDuration="13.717615688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:17.715613667 +0000 UTC m=+645.014091421" watchObservedRunningTime="2026-07-29 21:14:17.717615688 +0000 UTC m=+645.016093496"
Jul 29 21:14:17.717864 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:17.717798    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-259k4\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="30505586-f169-478d-828c-8e89c17011bd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-259k4"
Jul 29 21:14:17.733951 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.733890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc" podStartSLOduration=13.733866332 podStartE2EDuration="13.733866332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:17.733518929 +0000 UTC m=+645.031996679" watchObservedRunningTime="2026-07-29 21:14:17.733866332 +0000 UTC m=+645.032344065"
Jul 29 21:14:17.735509 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:17.735475    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fw5h9\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="ce8ca9e0-86ee-4891-b311-a4d9034e3955" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fw5h9"
Jul 29 21:14:17.756862 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.756393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" podStartSLOduration=13.756367986 podStartE2EDuration="13.756367986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:17.754514323 +0000 UTC m=+645.052992079" watchObservedRunningTime="2026-07-29 21:14:17.756367986 +0000 UTC m=+645.054845742"
Jul 29 21:14:17.768866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:17.768813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zfkl8" podStartSLOduration=1.634482784 podStartE2EDuration="23.768794502s" totalImagesPullingTime="22.134311718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:17.768324424 +0000 UTC m=+645.066802180" watchObservedRunningTime="2026-07-29 21:14:17.768794502 +0000 UTC m=+645.067272258"
Jul 29 21:14:19.198200 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:19.198159    2581 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-95rw8_kuadrant-system_9609f211-839e-4edd-aced-a17284c0f375_0(2245482fe7c9e83e3cc4fa87588f817bb65b956f6c0234fefb2044610749866e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-95rw8 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 29 21:14:19.198475 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:19.198233    2581 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-95rw8_kuadrant-system_9609f211-839e-4edd-aced-a17284c0f375_0(2245482fe7c9e83e3cc4fa87588f817bb65b956f6c0234fefb2044610749866e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-95rw8 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:19.715342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.715313    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:19.720945 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:19.718608    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="9609f211-839e-4edd-aced-a17284c0f375" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:19.729201 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.729172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:19.732056 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:19.732019    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="9609f211-839e-4edd-aced-a17284c0f375" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:19.763407 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.763377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume\") pod \"9609f211-839e-4edd-aced-a17284c0f375\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") "
Jul 29 21:14:19.763551 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.763478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb4k8\" (UniqueName: \"kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8\") pod \"9609f211-839e-4edd-aced-a17284c0f375\" (UID: \"9609f211-839e-4edd-aced-a17284c0f375\") "
Jul 29 21:14:19.763747 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.763723    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9609f211-839e-4edd-aced-a17284c0f375" (UID: "9609f211-839e-4edd-aced-a17284c0f375"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:19.765832 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.765808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8" (OuterVolumeSpecName: "kube-api-access-jb4k8") pod "9609f211-839e-4edd-aced-a17284c0f375" (UID: "9609f211-839e-4edd-aced-a17284c0f375"). InnerVolumeSpecName "kube-api-access-jb4k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:19.865168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.865124    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9609f211-839e-4edd-aced-a17284c0f375-extensions-socket-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:19.865168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:19.865157    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4k8\" (UniqueName: \"kubernetes.io/projected/9609f211-839e-4edd-aced-a17284c0f375-kube-api-access-jb4k8\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:20.720879 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:20.720842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:20.723196 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:20.723168    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="9609f211-839e-4edd-aced-a17284c0f375" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:20.728632 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:20.728590    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-95rw8\" is forbidden: User \"system:node:ip-10-0-143-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-168.ec2.internal' and this object" podUID="9609f211-839e-4edd-aced-a17284c0f375" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-95rw8"
Jul 29 21:14:21.186510 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:21.186475    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9609f211-839e-4edd-aced-a17284c0f375" path="/var/lib/kubelet/pods/9609f211-839e-4edd-aced-a17284c0f375/volumes"
Jul 29 21:14:28.713650 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:28.713617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m5szc"
Jul 29 21:14:28.714105 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:28.713713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:28.714105 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:28.713761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:28.797457 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:28.797413    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:28.797676 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:28.797648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" podUID="a70f1416-ab1e-415d-8b30-beff882874aa" containerName="manager" containerID="cri-o://925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3" gracePeriod=10
Jul 29 21:14:29.033541 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.033502    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:14:29.037074 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.037049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.053074 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.053043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:14:29.061873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.061841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:29.148490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.148458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mntv\" (UniqueName: \"kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv\") pod \"a70f1416-ab1e-415d-8b30-beff882874aa\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") "
Jul 29 21:14:29.148651 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.148561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume\") pod \"a70f1416-ab1e-415d-8b30-beff882874aa\" (UID: \"a70f1416-ab1e-415d-8b30-beff882874aa\") "
Jul 29 21:14:29.148752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.148730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.148805 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.148780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtm98\" (UniqueName: \"kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.149000 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.148983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a70f1416-ab1e-415d-8b30-beff882874aa" (UID: "a70f1416-ab1e-415d-8b30-beff882874aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:29.150628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.150604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv" (OuterVolumeSpecName: "kube-api-access-9mntv") pod "a70f1416-ab1e-415d-8b30-beff882874aa" (UID: "a70f1416-ab1e-415d-8b30-beff882874aa"). InnerVolumeSpecName "kube-api-access-9mntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:29.249854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.249756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.249854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.249800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtm98\" (UniqueName: \"kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.249854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.249834    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mntv\" (UniqueName: \"kubernetes.io/projected/a70f1416-ab1e-415d-8b30-beff882874aa-kube-api-access-9mntv\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:29.249854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.249845    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70f1416-ab1e-415d-8b30-beff882874aa-extensions-socket-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:29.250180 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.250153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.279601 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.279571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtm98\" (UniqueName: \"kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98\") pod \"kuadrant-operator-controller-manager-86878c769f-2nh96\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.357522 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.357486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.499632 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.499602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:14:29.502510 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:29.502482    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59880b_d47f_4961_a4bd_931a22415090.slice/crio-ab5e6a1ad270169e46f6e489491a1e5d5a7455ac0f2587fe3c42fc20cec308a4 WatchSource:0}: Error finding container ab5e6a1ad270169e46f6e489491a1e5d5a7455ac0f2587fe3c42fc20cec308a4: Status 404 returned error can't find the container with id ab5e6a1ad270169e46f6e489491a1e5d5a7455ac0f2587fe3c42fc20cec308a4
Jul 29 21:14:29.768484 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.768401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" event={"ID":"5e59880b-d47f-4961-a4bd-931a22415090","Type":"ContainerStarted","Data":"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"}
Jul 29 21:14:29.768484 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.768458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" event={"ID":"5e59880b-d47f-4961-a4bd-931a22415090","Type":"ContainerStarted","Data":"ab5e6a1ad270169e46f6e489491a1e5d5a7455ac0f2587fe3c42fc20cec308a4"}
Jul 29 21:14:29.768967 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.768516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:29.769998 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.769964    2581 generic.go:356] "Generic (PLEG): container finished" podID="a70f1416-ab1e-415d-8b30-beff882874aa" containerID="925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3" exitCode=0
Jul 29 21:14:29.770139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.770012    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"
Jul 29 21:14:29.770139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.770029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" event={"ID":"a70f1416-ab1e-415d-8b30-beff882874aa","Type":"ContainerDied","Data":"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"}
Jul 29 21:14:29.770139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.770075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46" event={"ID":"a70f1416-ab1e-415d-8b30-beff882874aa","Type":"ContainerDied","Data":"546c41f52b281483f54ffe597604aa93ef595585dcffafc51f09467c29d22c1d"}
Jul 29 21:14:29.770139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.770099    2581 scope.go:117] "RemoveContainer" containerID="925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"
Jul 29 21:14:29.779740 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.779687    2581 scope.go:117] "RemoveContainer" containerID="925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"
Jul 29 21:14:29.780093 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:29.780072    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3\": container with ID starting with 925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3 not found: ID does not exist" containerID="925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"
Jul 29 21:14:29.780150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.780105    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3"} err="failed to get container status \"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3\": rpc error: code = NotFound desc = could not find container \"925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3\": container with ID starting with 925ffe3db140f2889bf75c18a71efda4768cb15700b3567c4e0ffdfb3e45f1b3 not found: ID does not exist"
Jul 29 21:14:29.790166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.790107    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" podStartSLOduration=0.790089147 podStartE2EDuration="790.089147ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:14:29.788013203 +0000 UTC m=+657.086490957" watchObservedRunningTime="2026-07-29 21:14:29.790089147 +0000 UTC m=+657.088566901"
Jul 29 21:14:29.806517 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.806482    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:29.813218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:29.813190    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bpr46"]
Jul 29 21:14:31.186823 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:31.186786    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70f1416-ab1e-415d-8b30-beff882874aa" path="/var/lib/kubelet/pods/a70f1416-ab1e-415d-8b30-beff882874aa/volumes"
Jul 29 21:14:40.782679 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:40.782646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:14:40.832468 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:40.832429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:40.832844 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:40.832755    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" podUID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" containerName="manager" containerID="cri-o://e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec" gracePeriod=10
Jul 29 21:14:41.161603 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.161565    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:41.256547 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.256515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw7ld\" (UniqueName: \"kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld\") pod \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") "
Jul 29 21:14:41.256765 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.256558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume\") pod \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\" (UID: \"7687cf70-deda-4ce1-8dd4-71805e12e3fa\") "
Jul 29 21:14:41.256976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.256950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7687cf70-deda-4ce1-8dd4-71805e12e3fa" (UID: "7687cf70-deda-4ce1-8dd4-71805e12e3fa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:41.258817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.258798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld" (OuterVolumeSpecName: "kube-api-access-xw7ld") pod "7687cf70-deda-4ce1-8dd4-71805e12e3fa" (UID: "7687cf70-deda-4ce1-8dd4-71805e12e3fa"). InnerVolumeSpecName "kube-api-access-xw7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:41.357603 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.357561    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7687cf70-deda-4ce1-8dd4-71805e12e3fa-extensions-socket-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:41.357603 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.357603    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7ld\" (UniqueName: \"kubernetes.io/projected/7687cf70-deda-4ce1-8dd4-71805e12e3fa-kube-api-access-xw7ld\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:14:41.840368 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.840284    2581 generic.go:356] "Generic (PLEG): container finished" podID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" containerID="e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec" exitCode=0
Jul 29 21:14:41.840368 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.840344    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"
Jul 29 21:14:41.840909 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.840350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" event={"ID":"7687cf70-deda-4ce1-8dd4-71805e12e3fa","Type":"ContainerDied","Data":"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"}
Jul 29 21:14:41.840909 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.840476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz" event={"ID":"7687cf70-deda-4ce1-8dd4-71805e12e3fa","Type":"ContainerDied","Data":"1d5b5fb48a5b336be39a801556d66f54661df7488e1c8505cbbd38318231d33f"}
Jul 29 21:14:41.840909 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.840512    2581 scope.go:117] "RemoveContainer" containerID="e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"
Jul 29 21:14:41.849851 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.849832    2581 scope.go:117] "RemoveContainer" containerID="e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"
Jul 29 21:14:41.850132 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:14:41.850110    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec\": container with ID starting with e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec not found: ID does not exist" containerID="e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"
Jul 29 21:14:41.850223 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.850141    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec"} err="failed to get container status \"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec\": rpc error: code = NotFound desc = could not find container \"e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec\": container with ID starting with e9e772fd3278ff85991d73a673f630c254657cc65f51a16f7410920d405b21ec not found: ID does not exist"
Jul 29 21:14:41.868409 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.868377    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:41.872840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:41.872815    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6g5nz"]
Jul 29 21:14:43.187212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:43.187170    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" path="/var/lib/kubelet/pods/7687cf70-deda-4ce1-8dd4-71805e12e3fa/volumes"
Jul 29 21:14:58.109960 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.109919    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110250    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" containerName="manager"
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110261    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" containerName="manager"
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110278    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70f1416-ab1e-415d-8b30-beff882874aa" containerName="manager"
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110285    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70f1416-ab1e-415d-8b30-beff882874aa" containerName="manager"
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110335    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7687cf70-deda-4ce1-8dd4-71805e12e3fa" containerName="manager"
Jul 29 21:14:58.110367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.110342    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70f1416-ab1e-415d-8b30-beff882874aa" containerName="manager"
Jul 29 21:14:58.113205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.113188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.115682 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.115663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 21:14:58.120915 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.120891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:14:58.189251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.189214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.189413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.189267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gl9v\" (UniqueName: \"kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.200366 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.200334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:14:58.290184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.290146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.290184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.290188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gl9v\" (UniqueName: \"kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.290831 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.290812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.298045 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.298017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gl9v\" (UniqueName: \"kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v\") pod \"limitador-limitador-64c94fd64f-sxm5b\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.425931 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.425833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:14:58.566768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.566741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:14:58.568746 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:58.568711    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b294357_d092_4296_9963_5a8ab1dec571.slice/crio-07e8a77e35dc1247783564870bd283f0a43fc0f3862867efa9c877ffd9e154d0 WatchSource:0}: Error finding container 07e8a77e35dc1247783564870bd283f0a43fc0f3862867efa9c877ffd9e154d0: Status 404 returned error can't find the container with id 07e8a77e35dc1247783564870bd283f0a43fc0f3862867efa9c877ffd9e154d0
Jul 29 21:14:58.890509 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.890477    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:14:58.894982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.894953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:14:58.897150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.897126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dzhbh\""
Jul 29 21:14:58.900049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.900020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:14:58.928206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.928170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" event={"ID":"5b294357-d092-4296-9963-5a8ab1dec571","Type":"ContainerStarted","Data":"07e8a77e35dc1247783564870bd283f0a43fc0f3862867efa9c877ffd9e154d0"}
Jul 29 21:14:58.995969 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:58.995933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjb9q\" (UniqueName: \"kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q\") pod \"authorino-58cd955675-9w7j8\" (UID: \"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5\") " pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:14:59.068644 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.068606    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:14:59.071853 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.071832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:14:59.075894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.075857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:14:59.096894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.096857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzgl\" (UniqueName: \"kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl\") pod \"authorino-75d556bd77-pz6wc\" (UID: \"fa9e679c-068e-42c4-89f1-88483054b722\") " pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:14:59.097044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.096977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjb9q\" (UniqueName: \"kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q\") pod \"authorino-58cd955675-9w7j8\" (UID: \"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5\") " pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:14:59.107120 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.107092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjb9q\" (UniqueName: \"kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q\") pod \"authorino-58cd955675-9w7j8\" (UID: \"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5\") " pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:14:59.197657 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.197582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzgl\" (UniqueName: \"kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl\") pod \"authorino-75d556bd77-pz6wc\" (UID: \"fa9e679c-068e-42c4-89f1-88483054b722\") " pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:14:59.206014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.205978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzgl\" (UniqueName: \"kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl\") pod \"authorino-75d556bd77-pz6wc\" (UID: \"fa9e679c-068e-42c4-89f1-88483054b722\") " pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:14:59.206605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.206586    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:14:59.382169 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.382131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:14:59.391968 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.391894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:14:59.405090 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:59.396170    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b7bf3b_88a7_4c44_9650_f84b1e8fcee5.slice/crio-dcbe668fff50bec3466b519f9f5562f88c5b62285129601495f5b72cab7afe35 WatchSource:0}: Error finding container dcbe668fff50bec3466b519f9f5562f88c5b62285129601495f5b72cab7afe35: Status 404 returned error can't find the container with id dcbe668fff50bec3466b519f9f5562f88c5b62285129601495f5b72cab7afe35
Jul 29 21:14:59.579929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.579895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:14:59.582356 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:14:59.582319    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa9e679c_068e_42c4_89f1_88483054b722.slice/crio-babdf38e9869a6194e4eabe0865aec19cc9e0cad63d50f49aa339f2a8b51d16b WatchSource:0}: Error finding container babdf38e9869a6194e4eabe0865aec19cc9e0cad63d50f49aa339f2a8b51d16b: Status 404 returned error can't find the container with id babdf38e9869a6194e4eabe0865aec19cc9e0cad63d50f49aa339f2a8b51d16b
Jul 29 21:14:59.935656 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.935621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pz6wc" event={"ID":"fa9e679c-068e-42c4-89f1-88483054b722","Type":"ContainerStarted","Data":"babdf38e9869a6194e4eabe0865aec19cc9e0cad63d50f49aa339f2a8b51d16b"}
Jul 29 21:14:59.937017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:14:59.936987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w7j8" event={"ID":"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5","Type":"ContainerStarted","Data":"dcbe668fff50bec3466b519f9f5562f88c5b62285129601495f5b72cab7afe35"}
Jul 29 21:15:03.964802 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.964759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pz6wc" event={"ID":"fa9e679c-068e-42c4-89f1-88483054b722","Type":"ContainerStarted","Data":"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"}
Jul 29 21:15:03.966356 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.966324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" event={"ID":"5b294357-d092-4296-9963-5a8ab1dec571","Type":"ContainerStarted","Data":"f3ae6ab8189861034d41425727b569e3faeeee5a8195fa70c03b577bcb4f9938"}
Jul 29 21:15:03.966506 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.966384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:15:03.967843 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.967818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w7j8" event={"ID":"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5","Type":"ContainerStarted","Data":"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"}
Jul 29 21:15:03.980623 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.980570    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pz6wc" podStartSLOduration=1.174122374 podStartE2EDuration="4.980551935s" totalImagesPullingTime="3.806429561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:03.978815682 +0000 UTC m=+691.277293439" watchObservedRunningTime="2026-07-29 21:15:03.980551935 +0000 UTC m=+691.279029690"
Jul 29 21:15:03.992974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:03.992922    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9w7j8" podStartSLOduration=2.004715786 podStartE2EDuration="5.992900727s" totalImagesPullingTime="3.988184941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:03.992041825 +0000 UTC m=+691.290519586" watchObservedRunningTime="2026-07-29 21:15:03.992900727 +0000 UTC m=+691.291378515"
Jul 29 21:15:04.009595 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:04.009539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" podStartSLOduration=1.193474932 podStartE2EDuration="6.009522928s" totalImagesPullingTime="4.816047996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:04.007343528 +0000 UTC m=+691.305821332" watchObservedRunningTime="2026-07-29 21:15:04.009522928 +0000 UTC m=+691.308000682"
Jul 29 21:15:04.015000 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:04.014967    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:15:05.977943 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:05.977895    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9w7j8" podUID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" containerName="authorino" containerID="cri-o://717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac" gracePeriod=30
Jul 29 21:15:06.235423 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.235352    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:15:06.264749 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.264712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjb9q\" (UniqueName: \"kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q\") pod \"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5\" (UID: \"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5\") "
Jul 29 21:15:06.267027 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.266992    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q" (OuterVolumeSpecName: "kube-api-access-gjb9q") pod "44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" (UID: "44b7bf3b-88a7-4c44-9650-f84b1e8fcee5"). InnerVolumeSpecName "kube-api-access-gjb9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:06.365681 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.365636    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjb9q\" (UniqueName: \"kubernetes.io/projected/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5-kube-api-access-gjb9q\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:06.983826 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.983748    2581 generic.go:356] "Generic (PLEG): container finished" podID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" containerID="717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac" exitCode=0
Jul 29 21:15:06.983826 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.983799    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w7j8"
Jul 29 21:15:06.984376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.983836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w7j8" event={"ID":"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5","Type":"ContainerDied","Data":"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"}
Jul 29 21:15:06.984376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.983886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w7j8" event={"ID":"44b7bf3b-88a7-4c44-9650-f84b1e8fcee5","Type":"ContainerDied","Data":"dcbe668fff50bec3466b519f9f5562f88c5b62285129601495f5b72cab7afe35"}
Jul 29 21:15:06.984376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.983908    2581 scope.go:117] "RemoveContainer" containerID="717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"
Jul 29 21:15:06.993049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.993031    2581 scope.go:117] "RemoveContainer" containerID="717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"
Jul 29 21:15:06.993361 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:15:06.993341    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac\": container with ID starting with 717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac not found: ID does not exist" containerID="717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"
Jul 29 21:15:06.993428 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:06.993374    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac"} err="failed to get container status \"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac\": rpc error: code = NotFound desc = could not find container \"717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac\": container with ID starting with 717c80c9cd089c254cad0902d2ed14228b76f306551c7966193d79dc0fe996ac not found: ID does not exist"
Jul 29 21:15:07.005456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:07.005419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:15:07.009243 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:07.009215    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w7j8"]
Jul 29 21:15:07.187122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:07.187080    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" path="/var/lib/kubelet/pods/44b7bf3b-88a7-4c44-9650-f84b1e8fcee5/volumes"
Jul 29 21:15:13.223093 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:13.222786    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:15:13.223660 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:13.223158    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" podUID="5b294357-d092-4296-9963-5a8ab1dec571" containerName="limitador" containerID="cri-o://f3ae6ab8189861034d41425727b569e3faeeee5a8195fa70c03b577bcb4f9938" gracePeriod=30
Jul 29 21:15:13.228655 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:13.228631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:15:14.020797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.020762    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b294357-d092-4296-9963-5a8ab1dec571" containerID="f3ae6ab8189861034d41425727b569e3faeeee5a8195fa70c03b577bcb4f9938" exitCode=0
Jul 29 21:15:14.020970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.020842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" event={"ID":"5b294357-d092-4296-9963-5a8ab1dec571","Type":"ContainerDied","Data":"f3ae6ab8189861034d41425727b569e3faeeee5a8195fa70c03b577bcb4f9938"}
Jul 29 21:15:14.174299 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.174274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:15:14.229412 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.229369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gl9v\" (UniqueName: \"kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v\") pod \"5b294357-d092-4296-9963-5a8ab1dec571\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") "
Jul 29 21:15:14.229821 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.229434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file\") pod \"5b294357-d092-4296-9963-5a8ab1dec571\" (UID: \"5b294357-d092-4296-9963-5a8ab1dec571\") "
Jul 29 21:15:14.229892 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.229866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file" (OuterVolumeSpecName: "config-file") pod "5b294357-d092-4296-9963-5a8ab1dec571" (UID: "5b294357-d092-4296-9963-5a8ab1dec571"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:15:14.231632 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.231602    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v" (OuterVolumeSpecName: "kube-api-access-7gl9v") pod "5b294357-d092-4296-9963-5a8ab1dec571" (UID: "5b294357-d092-4296-9963-5a8ab1dec571"). InnerVolumeSpecName "kube-api-access-7gl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:14.330978 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.330886    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b294357-d092-4296-9963-5a8ab1dec571-config-file\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:14.330978 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:14.330939    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gl9v\" (UniqueName: \"kubernetes.io/projected/5b294357-d092-4296-9963-5a8ab1dec571-kube-api-access-7gl9v\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:15.027044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.027011    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"
Jul 29 21:15:15.027299 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.027011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-sxm5b" event={"ID":"5b294357-d092-4296-9963-5a8ab1dec571","Type":"ContainerDied","Data":"07e8a77e35dc1247783564870bd283f0a43fc0f3862867efa9c877ffd9e154d0"}
Jul 29 21:15:15.027299 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.027144    2581 scope.go:117] "RemoveContainer" containerID="f3ae6ab8189861034d41425727b569e3faeeee5a8195fa70c03b577bcb4f9938"
Jul 29 21:15:15.048926 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.048895    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:15:15.052206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.052181    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-sxm5b"]
Jul 29 21:15:15.186987 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:15.186951    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b294357-d092-4296-9963-5a8ab1dec571" path="/var/lib/kubelet/pods/5b294357-d092-4296-9963-5a8ab1dec571/volumes"
Jul 29 21:15:19.369183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369149    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"]
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369451    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b294357-d092-4296-9963-5a8ab1dec571" containerName="limitador"
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369462    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b294357-d092-4296-9963-5a8ab1dec571" containerName="limitador"
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" containerName="authorino"
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369481    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" containerName="authorino"
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369538    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b7bf3b-88a7-4c44-9650-f84b1e8fcee5" containerName="authorino"
Jul 29 21:15:19.369668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.369547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b294357-d092-4296-9963-5a8ab1dec571" containerName="limitador"
Jul 29 21:15:19.373886 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.373864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.378179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.378149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 21:15:19.379512 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.379051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:15:19.379512 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.379296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jdbjw\""
Jul 29 21:15:19.379512 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.379413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 21:15:19.381166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.381142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"]
Jul 29 21:15:19.472883 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.472842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/443b3da5-ce8f-41ff-bef9-439e20baef03-data\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.473065 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.472904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7w8\" (UniqueName: \"kubernetes.io/projected/443b3da5-ce8f-41ff-bef9-439e20baef03-kube-api-access-sp7w8\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.573608 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.573565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/443b3da5-ce8f-41ff-bef9-439e20baef03-data\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.573872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.573622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7w8\" (UniqueName: \"kubernetes.io/projected/443b3da5-ce8f-41ff-bef9-439e20baef03-kube-api-access-sp7w8\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.573995 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.573975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/443b3da5-ce8f-41ff-bef9-439e20baef03-data\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.587267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.587227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7w8\" (UniqueName: \"kubernetes.io/projected/443b3da5-ce8f-41ff-bef9-439e20baef03-kube-api-access-sp7w8\") pod \"postgres-69fb4594cf-tmk4c\" (UID: \"443b3da5-ce8f-41ff-bef9-439e20baef03\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.686301 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.686213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:19.841250 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:19.841218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"]
Jul 29 21:15:19.843839 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:15:19.843809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443b3da5_ce8f_41ff_bef9_439e20baef03.slice/crio-8a1c040ed06da933e991fe4900d45a13b5e6668aacb9b40e28153bfb5e728be4 WatchSource:0}: Error finding container 8a1c040ed06da933e991fe4900d45a13b5e6668aacb9b40e28153bfb5e728be4: Status 404 returned error can't find the container with id 8a1c040ed06da933e991fe4900d45a13b5e6668aacb9b40e28153bfb5e728be4
Jul 29 21:15:20.055684 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:20.055597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c" event={"ID":"443b3da5-ce8f-41ff-bef9-439e20baef03","Type":"ContainerStarted","Data":"8a1c040ed06da933e991fe4900d45a13b5e6668aacb9b40e28153bfb5e728be4"}
Jul 29 21:15:25.090708 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:25.090594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c" event={"ID":"443b3da5-ce8f-41ff-bef9-439e20baef03","Type":"ContainerStarted","Data":"6f782cf67f9fb385241b92b02fecb618c5aff65f416b8accbbd09b0bf90266d8"}
Jul 29 21:15:25.091081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:25.090778    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:25.106228 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:25.106173    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c" podStartSLOduration=1.26645824 podStartE2EDuration="6.106155645s" totalImagesPullingTime="4.839697405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:25.104504959 +0000 UTC m=+712.402982715" watchObservedRunningTime="2026-07-29 21:15:25.106155645 +0000 UTC m=+712.404633399"
Jul 29 21:15:31.128029 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.127999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tmk4c"
Jul 29 21:15:31.696977 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.696929    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kbpcv"]
Jul 29 21:15:31.704554 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.704522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:31.705867 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.705834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kbpcv"]
Jul 29 21:15:31.779760 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.779686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44nx\" (UniqueName: \"kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx\") pod \"authorino-5d4b67c446-kbpcv\" (UID: \"3da6e38a-29b9-4aaa-8181-ad84b962c2de\") " pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:31.880437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.880393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44nx\" (UniqueName: \"kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx\") pod \"authorino-5d4b67c446-kbpcv\" (UID: \"3da6e38a-29b9-4aaa-8181-ad84b962c2de\") " pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:31.883866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.883824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kbpcv"]
Jul 29 21:15:31.884106 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:15:31.884083    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s44nx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-kbpcv" podUID="3da6e38a-29b9-4aaa-8181-ad84b962c2de"
Jul 29 21:15:31.891586 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.891558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44nx\" (UniqueName: \"kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx\") pod \"authorino-5d4b67c446-kbpcv\" (UID: \"3da6e38a-29b9-4aaa-8181-ad84b962c2de\") " pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:31.911466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.911425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-bdstf"]
Jul 29 21:15:31.915292 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.915273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:31.917661 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.917635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 21:15:31.925058 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.925030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-bdstf"]
Jul 29 21:15:31.945830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.945797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-bdstf"]
Jul 29 21:15:31.946059 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:15:31.946039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ctfp5 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-bdstf" podUID="57464ccf-48bc-45b8-9f9c-38d211dcc746"
Jul 29 21:15:31.981143 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.981048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfp5\" (UniqueName: \"kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:31.981143 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.981137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:31.983078 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.983052    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:15:31.986887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.986863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:31.998306 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:31.998279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:15:32.082250 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.082203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfp5\" (UniqueName: \"kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.082440 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.082269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.082440 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.082428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.082509 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.082460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jwc\" (UniqueName: \"kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.084920 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.084892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.090562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.090536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfp5\" (UniqueName: \"kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5\") pod \"authorino-76877994-bdstf\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") " pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.127552 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.127518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:32.127751 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.127518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.135469 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.135443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:32.138647 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.138623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:32.183686 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.183649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.183686 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.183687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jwc\" (UniqueName: \"kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.186445 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.186417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.192032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.192003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jwc\" (UniqueName: \"kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc\") pod \"authorino-bd49d8576-bdn7m\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") " pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.285126 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.285033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctfp5\" (UniqueName: \"kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5\") pod \"57464ccf-48bc-45b8-9f9c-38d211dcc746\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") "
Jul 29 21:15:32.285126 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.285075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert\") pod \"57464ccf-48bc-45b8-9f9c-38d211dcc746\" (UID: \"57464ccf-48bc-45b8-9f9c-38d211dcc746\") "
Jul 29 21:15:32.285328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.285135    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s44nx\" (UniqueName: \"kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx\") pod \"3da6e38a-29b9-4aaa-8181-ad84b962c2de\" (UID: \"3da6e38a-29b9-4aaa-8181-ad84b962c2de\") "
Jul 29 21:15:32.287587 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.287555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx" (OuterVolumeSpecName: "kube-api-access-s44nx") pod "3da6e38a-29b9-4aaa-8181-ad84b962c2de" (UID: "3da6e38a-29b9-4aaa-8181-ad84b962c2de"). InnerVolumeSpecName "kube-api-access-s44nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:32.287873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.287857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5" (OuterVolumeSpecName: "kube-api-access-ctfp5") pod "57464ccf-48bc-45b8-9f9c-38d211dcc746" (UID: "57464ccf-48bc-45b8-9f9c-38d211dcc746"). InnerVolumeSpecName "kube-api-access-ctfp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:32.287915 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.287875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "57464ccf-48bc-45b8-9f9c-38d211dcc746" (UID: "57464ccf-48bc-45b8-9f9c-38d211dcc746"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:15:32.296080 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.296043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:15:32.386439 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.386396    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfp5\" (UniqueName: \"kubernetes.io/projected/57464ccf-48bc-45b8-9f9c-38d211dcc746-kube-api-access-ctfp5\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:32.386439 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.386439    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57464ccf-48bc-45b8-9f9c-38d211dcc746-tls-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:32.386439 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.386454    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44nx\" (UniqueName: \"kubernetes.io/projected/3da6e38a-29b9-4aaa-8181-ad84b962c2de-kube-api-access-s44nx\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:32.437982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:32.437951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:15:32.439900 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:15:32.439871    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c491116_442a_48f1_895b_e7a567880aae.slice/crio-2a12104fc011a78cbc3db2c64a2d4b032a36b5e07797ed8471e7fdf98f41e711 WatchSource:0}: Error finding container 2a12104fc011a78cbc3db2c64a2d4b032a36b5e07797ed8471e7fdf98f41e711: Status 404 returned error can't find the container with id 2a12104fc011a78cbc3db2c64a2d4b032a36b5e07797ed8471e7fdf98f41e711
Jul 29 21:15:33.135072 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.135040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd49d8576-bdn7m" event={"ID":"1c491116-442a-48f1-895b-e7a567880aae","Type":"ContainerStarted","Data":"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"}
Jul 29 21:15:33.135178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.135089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd49d8576-bdn7m" event={"ID":"1c491116-442a-48f1-895b-e7a567880aae","Type":"ContainerStarted","Data":"2a12104fc011a78cbc3db2c64a2d4b032a36b5e07797ed8471e7fdf98f41e711"}
Jul 29 21:15:33.135215 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.135188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kbpcv"
Jul 29 21:15:33.135289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.135273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bdstf"
Jul 29 21:15:33.150497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.150447    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bd49d8576-bdn7m" podStartSLOduration=1.757812258 podStartE2EDuration="2.150429318s" totalImagesPullingTime="392.61706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:33.148564912 +0000 UTC m=+720.447042667" watchObservedRunningTime="2026-07-29 21:15:33.150429318 +0000 UTC m=+720.448907073"
Jul 29 21:15:33.173124 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.173079    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kbpcv"]
Jul 29 21:15:33.193892 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.193856    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:15:33.194075 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.193933    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kbpcv"]
Jul 29 21:15:33.194460 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.194405    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pz6wc" podUID="fa9e679c-068e-42c4-89f1-88483054b722" containerName="authorino" containerID="cri-o://f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488" gracePeriod=30
Jul 29 21:15:33.207771 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.207738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-bdstf"]
Jul 29 21:15:33.212229 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.212198    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-bdstf"]
Jul 29 21:15:33.488163 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.488130    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:15:33.596674 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.596632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfzgl\" (UniqueName: \"kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl\") pod \"fa9e679c-068e-42c4-89f1-88483054b722\" (UID: \"fa9e679c-068e-42c4-89f1-88483054b722\") "
Jul 29 21:15:33.599142 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.599108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl" (OuterVolumeSpecName: "kube-api-access-dfzgl") pod "fa9e679c-068e-42c4-89f1-88483054b722" (UID: "fa9e679c-068e-42c4-89f1-88483054b722"). InnerVolumeSpecName "kube-api-access-dfzgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:33.697688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:33.697600    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzgl\" (UniqueName: \"kubernetes.io/projected/fa9e679c-068e-42c4-89f1-88483054b722-kube-api-access-dfzgl\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:34.142367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.142325    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa9e679c-068e-42c4-89f1-88483054b722" containerID="f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488" exitCode=0
Jul 29 21:15:34.142524 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.142371    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pz6wc"
Jul 29 21:15:34.142524 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.142415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pz6wc" event={"ID":"fa9e679c-068e-42c4-89f1-88483054b722","Type":"ContainerDied","Data":"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"}
Jul 29 21:15:34.142524 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.142460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pz6wc" event={"ID":"fa9e679c-068e-42c4-89f1-88483054b722","Type":"ContainerDied","Data":"babdf38e9869a6194e4eabe0865aec19cc9e0cad63d50f49aa339f2a8b51d16b"}
Jul 29 21:15:34.142524 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.142484    2581 scope.go:117] "RemoveContainer" containerID="f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"
Jul 29 21:15:34.152507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.152482    2581 scope.go:117] "RemoveContainer" containerID="f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"
Jul 29 21:15:34.152881 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:15:34.152854    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488\": container with ID starting with f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488 not found: ID does not exist" containerID="f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"
Jul 29 21:15:34.152935 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.152900    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488"} err="failed to get container status \"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488\": rpc error: code = NotFound desc = could not find container \"f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488\": container with ID starting with f6db1cf45c8ab3d92f32f0664e48b7f28b084d8e6e23baefb37e8749e4418488 not found: ID does not exist"
Jul 29 21:15:34.166111 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.166075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:15:34.169405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:34.169374    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pz6wc"]
Jul 29 21:15:35.187143 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:35.187109    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da6e38a-29b9-4aaa-8181-ad84b962c2de" path="/var/lib/kubelet/pods/3da6e38a-29b9-4aaa-8181-ad84b962c2de/volumes"
Jul 29 21:15:35.187544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:35.187368    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57464ccf-48bc-45b8-9f9c-38d211dcc746" path="/var/lib/kubelet/pods/57464ccf-48bc-45b8-9f9c-38d211dcc746/volumes"
Jul 29 21:15:35.187585 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:35.187546    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9e679c-068e-42c4-89f1-88483054b722" path="/var/lib/kubelet/pods/fa9e679c-068e-42c4-89f1-88483054b722/volumes"
Jul 29 21:15:38.011444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.011404    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:38.011921 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.011744    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9e679c-068e-42c4-89f1-88483054b722" containerName="authorino"
Jul 29 21:15:38.011921 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.011760    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9e679c-068e-42c4-89f1-88483054b722" containerName="authorino"
Jul 29 21:15:38.011921 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.011829    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9e679c-068e-42c4-89f1-88483054b722" containerName="authorino"
Jul 29 21:15:38.163146 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.163106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:38.163327 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.163181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.165474 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.165452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:15:38.165751 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.165735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:15:38.166390 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.166376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-slf4h\""
Jul 29 21:15:38.185415 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.185380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:15:38.205573 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.205533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:15:38.205753 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.205639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.237532 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.237492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fwb\" (UniqueName: \"kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.237723 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.237549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.338586 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.338508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fwb\" (UniqueName: \"kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.338586 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.338549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc85\" (UniqueName: \"kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.338586 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.338577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.338853 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.338596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.341201 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.341173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.348469 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.348441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fwb\" (UniqueName: \"kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb\") pod \"maas-controller-57759bf96f-krntl\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") " pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.439671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.439630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc85\" (UniqueName: \"kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.439671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.439678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.442262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.442240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.449242 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.449208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc85\" (UniqueName: \"kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85\") pod \"maas-controller-6469f67b9b-6hr9c\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") " pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.474055 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.474017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:38.515998 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.515952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:38.624898 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.624777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:38.629372 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:15:38.629322    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2976f00f_6455_4f2f_bbd9_8ab1d20c33a6.slice/crio-cb7b0357a49faac5142faf4713bd60bfceece26f9bf23af510a7b4d51bd24579 WatchSource:0}: Error finding container cb7b0357a49faac5142faf4713bd60bfceece26f9bf23af510a7b4d51bd24579: Status 404 returned error can't find the container with id cb7b0357a49faac5142faf4713bd60bfceece26f9bf23af510a7b4d51bd24579
Jul 29 21:15:38.673631 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:38.673587    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:15:38.676558 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:15:38.676528    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040ecb7d_0f30_410f_b623_690dfa00ecb2.slice/crio-9a99e2c415745847226d9e288f2d4a5dd3e997fa0c1f06636a6c6c35f7572997 WatchSource:0}: Error finding container 9a99e2c415745847226d9e288f2d4a5dd3e997fa0c1f06636a6c6c35f7572997: Status 404 returned error can't find the container with id 9a99e2c415745847226d9e288f2d4a5dd3e997fa0c1f06636a6c6c35f7572997
Jul 29 21:15:39.172889 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:39.172845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57759bf96f-krntl" event={"ID":"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6","Type":"ContainerStarted","Data":"cb7b0357a49faac5142faf4713bd60bfceece26f9bf23af510a7b4d51bd24579"}
Jul 29 21:15:39.174882 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:39.174846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" event={"ID":"040ecb7d-0f30-410f-b623-690dfa00ecb2","Type":"ContainerStarted","Data":"9a99e2c415745847226d9e288f2d4a5dd3e997fa0c1f06636a6c6c35f7572997"}
Jul 29 21:15:42.196578 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.196533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" event={"ID":"040ecb7d-0f30-410f-b623-690dfa00ecb2","Type":"ContainerStarted","Data":"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"}
Jul 29 21:15:42.197081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.196617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:42.198288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.198254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57759bf96f-krntl" event={"ID":"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6","Type":"ContainerStarted","Data":"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"}
Jul 29 21:15:42.198393 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.198318    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:42.212299 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.212247    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" podStartSLOduration=1.744844691 podStartE2EDuration="4.212230962s" totalImagesPullingTime="2.467386271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:42.211251388 +0000 UTC m=+729.509729612" watchObservedRunningTime="2026-07-29 21:15:42.212230962 +0000 UTC m=+729.510708716"
Jul 29 21:15:42.226986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:42.226929    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57759bf96f-krntl" podStartSLOduration=2.705368423 podStartE2EDuration="5.226911278s" totalImagesPullingTime="2.521542855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:15:42.225677849 +0000 UTC m=+729.524155605" watchObservedRunningTime="2026-07-29 21:15:42.226911278 +0000 UTC m=+729.525389032"
Jul 29 21:15:53.204677 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.204648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:15:53.205253 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.204946    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:53.264393 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.264350    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:53.264734 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.264654    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57759bf96f-krntl" podUID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" containerName="manager" containerID="cri-o://702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d" gracePeriod=30
Jul 29 21:15:53.513534 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.513508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:53.670185 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.670142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert\") pod \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") "
Jul 29 21:15:53.670393 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.670207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8fwb\" (UniqueName: \"kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb\") pod \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\" (UID: \"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6\") "
Jul 29 21:15:53.672622 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.672590    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb" (OuterVolumeSpecName: "kube-api-access-n8fwb") pod "2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" (UID: "2976f00f-6455-4f2f-bbd9-8ab1d20c33a6"). InnerVolumeSpecName "kube-api-access-n8fwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:15:53.672622 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.672604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" (UID: "2976f00f-6455-4f2f-bbd9-8ab1d20c33a6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:15:53.771376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.771285    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fwb\" (UniqueName: \"kubernetes.io/projected/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-kube-api-access-n8fwb\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:53.771376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:53.771319    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6-webhook-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:15:54.260186 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.260154    2581 generic.go:356] "Generic (PLEG): container finished" podID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" containerID="702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d" exitCode=0
Jul 29 21:15:54.260605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.260201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57759bf96f-krntl"
Jul 29 21:15:54.260605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.260237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57759bf96f-krntl" event={"ID":"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6","Type":"ContainerDied","Data":"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"}
Jul 29 21:15:54.260605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.260301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57759bf96f-krntl" event={"ID":"2976f00f-6455-4f2f-bbd9-8ab1d20c33a6","Type":"ContainerDied","Data":"cb7b0357a49faac5142faf4713bd60bfceece26f9bf23af510a7b4d51bd24579"}
Jul 29 21:15:54.260605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.260325    2581 scope.go:117] "RemoveContainer" containerID="702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"
Jul 29 21:15:54.270431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.270268    2581 scope.go:117] "RemoveContainer" containerID="702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"
Jul 29 21:15:54.270624 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:15:54.270597    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d\": container with ID starting with 702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d not found: ID does not exist" containerID="702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"
Jul 29 21:15:54.270670 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.270635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d"} err="failed to get container status \"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d\": rpc error: code = NotFound desc = could not find container \"702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d\": container with ID starting with 702eff761c9dd33664d596c15ae0a64bde3e26b54161bb35ad82c05970ea707d not found: ID does not exist"
Jul 29 21:15:54.281231 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.281200    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:54.284874 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:54.284847    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57759bf96f-krntl"]
Jul 29 21:15:55.186810 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:15:55.186771    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" path="/var/lib/kubelet/pods/2976f00f-6455-4f2f-bbd9-8ab1d20c33a6/volumes"
Jul 29 21:16:16.051688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.051650    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"]
Jul 29 21:16:16.052081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.052004    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" containerName="manager"
Jul 29 21:16:16.052081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.052015    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" containerName="manager"
Jul 29 21:16:16.052081 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.052067    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2976f00f-6455-4f2f-bbd9-8ab1d20c33a6" containerName="manager"
Jul 29 21:16:16.059550 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.059529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.061806 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.061788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 21:16:16.061919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.061885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-77t9w\""
Jul 29 21:16:16.065752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.065728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"]
Jul 29 21:16:16.134175 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.134142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.134352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.134197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-openshift-service-ca\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.134352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.134260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8l8\" (UniqueName: \"kubernetes.io/projected/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-kube-api-access-4g8l8\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.235685 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.235646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.235906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.235758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-openshift-service-ca\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.235906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.235808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8l8\" (UniqueName: \"kubernetes.io/projected/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-kube-api-access-4g8l8\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.235906 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:16:16.235841    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 29 21:16:16.236073 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:16:16.235928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls podName:2d0bdf8f-fefb-4f89-ac50-a2973b77fa54 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:16.735906957 +0000 UTC m=+764.034384692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls") pod "maas-api-5b5d9dbbcd-p5vtv" (UID: "2d0bdf8f-fefb-4f89-ac50-a2973b77fa54") : secret "maas-api-serving-cert" not found
Jul 29 21:16:16.236497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.236473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-openshift-service-ca\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.244244 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.244209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8l8\" (UniqueName: \"kubernetes.io/projected/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-kube-api-access-4g8l8\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.462804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.462755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rngfn"]
Jul 29 21:16:16.467020 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.466994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.471640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.471615    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 21:16:16.471809 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.471615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bj7td\""
Jul 29 21:16:16.478609 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.478580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rngfn"]
Jul 29 21:16:16.501053 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.501016    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"]
Jul 29 21:16:16.504740 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.504716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.511164 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.511135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"]
Jul 29 21:16:16.538879 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.538842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423ba19-ac0d-42e1-8948-77a5aa291ec5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.538879 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.538885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7kl\" (UniqueName: \"kubernetes.io/projected/9423ba19-ac0d-42e1-8948-77a5aa291ec5-kube-api-access-kd7kl\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.539101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.538951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28k4\" (UniqueName: \"kubernetes.io/projected/71c41b59-4611-4075-81e2-ae896e03ca9d-kube-api-access-w28k4\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.539101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.539015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c41b59-4611-4075-81e2-ae896e03ca9d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.639716 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.639651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423ba19-ac0d-42e1-8948-77a5aa291ec5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.639929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.639743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7kl\" (UniqueName: \"kubernetes.io/projected/9423ba19-ac0d-42e1-8948-77a5aa291ec5-kube-api-access-kd7kl\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.639929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.639791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28k4\" (UniqueName: \"kubernetes.io/projected/71c41b59-4611-4075-81e2-ae896e03ca9d-kube-api-access-w28k4\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.640050 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.639965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c41b59-4611-4075-81e2-ae896e03ca9d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.640976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.640938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423ba19-ac0d-42e1-8948-77a5aa291ec5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.643715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.641250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c41b59-4611-4075-81e2-ae896e03ca9d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.648897 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.648871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7kl\" (UniqueName: \"kubernetes.io/projected/9423ba19-ac0d-42e1-8948-77a5aa291ec5-kube-api-access-kd7kl\") pod \"payload-processing-5cc6cd444b-rngfn\" (UID: \"9423ba19-ac0d-42e1-8948-77a5aa291ec5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.649016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.648910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28k4\" (UniqueName: \"kubernetes.io/projected/71c41b59-4611-4075-81e2-ae896e03ca9d-kube-api-access-w28k4\") pod \"payload-pre-processing-687dbb6cf6-wj8hl\" (UID: \"71c41b59-4611-4075-81e2-ae896e03ca9d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.741087 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.740982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.743539 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.743509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0bdf8f-fefb-4f89-ac50-a2973b77fa54-maas-api-tls\") pod \"maas-api-5b5d9dbbcd-p5vtv\" (UID: \"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54\") " pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.779476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.779437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:16.816919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.816881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:16.932881 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.932853    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-rngfn"]
Jul 29 21:16:16.934983 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:16:16.934953    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9423ba19_ac0d_42e1_8948_77a5aa291ec5.slice/crio-f3a940908286c9f69f27ea0ba157fa5cdcf130b52cb1591c6e3a879dc5d9cece WatchSource:0}: Error finding container f3a940908286c9f69f27ea0ba157fa5cdcf130b52cb1591c6e3a879dc5d9cece: Status 404 returned error can't find the container with id f3a940908286c9f69f27ea0ba157fa5cdcf130b52cb1591c6e3a879dc5d9cece
Jul 29 21:16:16.971249 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.971212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:16.982794 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:16.982769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"]
Jul 29 21:16:16.985677 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:16:16.985649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c41b59_4611_4075_81e2_ae896e03ca9d.slice/crio-83b6eab7712c3e3baa93a78f797e9e17945837daac2b342fc0299d22087e5980 WatchSource:0}: Error finding container 83b6eab7712c3e3baa93a78f797e9e17945837daac2b342fc0299d22087e5980: Status 404 returned error can't find the container with id 83b6eab7712c3e3baa93a78f797e9e17945837daac2b342fc0299d22087e5980
Jul 29 21:16:17.123046 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:17.123014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"]
Jul 29 21:16:17.124929 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:16:17.124899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0bdf8f_fefb_4f89_ac50_a2973b77fa54.slice/crio-29cd9a6ffddbb242efd54f12e41dc7f1a34244634b2b61216d2b8f72d6e7d8a9 WatchSource:0}: Error finding container 29cd9a6ffddbb242efd54f12e41dc7f1a34244634b2b61216d2b8f72d6e7d8a9: Status 404 returned error can't find the container with id 29cd9a6ffddbb242efd54f12e41dc7f1a34244634b2b61216d2b8f72d6e7d8a9
Jul 29 21:16:17.388484 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:17.388437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn" event={"ID":"9423ba19-ac0d-42e1-8948-77a5aa291ec5","Type":"ContainerStarted","Data":"f3a940908286c9f69f27ea0ba157fa5cdcf130b52cb1591c6e3a879dc5d9cece"}
Jul 29 21:16:17.389886 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:17.389853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl" event={"ID":"71c41b59-4611-4075-81e2-ae896e03ca9d","Type":"ContainerStarted","Data":"83b6eab7712c3e3baa93a78f797e9e17945837daac2b342fc0299d22087e5980"}
Jul 29 21:16:17.391530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:17.391505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv" event={"ID":"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54","Type":"ContainerStarted","Data":"29cd9a6ffddbb242efd54f12e41dc7f1a34244634b2b61216d2b8f72d6e7d8a9"}
Jul 29 21:16:19.407129 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.407097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl" event={"ID":"71c41b59-4611-4075-81e2-ae896e03ca9d","Type":"ContainerStarted","Data":"1161730dc994b54dff7e0ea869d4620d76bce3fe4cb36f65417d768129603f0b"}
Jul 29 21:16:19.407628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.407171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:19.408947 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.408919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv" event={"ID":"2d0bdf8f-fefb-4f89-ac50-a2973b77fa54","Type":"ContainerStarted","Data":"5af056d8024f722a3aebe0f77a96e144cb721f3f093ad7ec9c10ef52e5d7d51c"}
Jul 29 21:16:19.409086 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.408997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:19.410642 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.410616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn" event={"ID":"9423ba19-ac0d-42e1-8948-77a5aa291ec5","Type":"ContainerStarted","Data":"da65eca76f7fc07dc96998871aa08cc1b61595aa39699b889faa2de75d02f8e1"}
Jul 29 21:16:19.410777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.410750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:19.423279 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.423218    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl" podStartSLOduration=1.170007501 podStartE2EDuration="3.423202174s" totalImagesPullingTime="2.253194673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:19.4215261 +0000 UTC m=+766.720003854" watchObservedRunningTime="2026-07-29 21:16:19.423202174 +0000 UTC m=+766.721679928"
Jul 29 21:16:19.437539 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.437483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv" podStartSLOduration=1.3285401829999999 podStartE2EDuration="3.43746336s" totalImagesPullingTime="2.108923177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:19.435755624 +0000 UTC m=+766.734233379" watchObservedRunningTime="2026-07-29 21:16:19.43746336 +0000 UTC m=+766.735941115"
Jul 29 21:16:19.452429 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:19.452372    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn" podStartSLOduration=1.154625379 podStartE2EDuration="3.452353873s" totalImagesPullingTime="2.297728494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:19.450865104 +0000 UTC m=+766.749342861" watchObservedRunningTime="2026-07-29 21:16:19.452353873 +0000 UTC m=+766.750831627"
Jul 29 21:16:25.423592 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:25.423560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b5d9dbbcd-p5vtv"
Jul 29 21:16:30.419887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:30.419855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-rngfn"
Jul 29 21:16:30.420295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:30.419927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj8hl"
Jul 29 21:16:38.173896 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.173854    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"]
Jul 29 21:16:38.179196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.179159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.181478 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.181454    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 21:16:38.182503 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.182477    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 21:16:38.182627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.182536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gqldj\""
Jul 29 21:16:38.182627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.182536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 21:16:38.190079 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.190028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"]
Jul 29 21:16:38.235066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.235066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3043ba-32b5-4c15-bb95-4569c88fff70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.235324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsz5\" (UniqueName: \"kubernetes.io/projected/fc3043ba-32b5-4c15-bb95-4569c88fff70-kube-api-access-kxsz5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.235324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.235324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.235324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.235295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.335965 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.335930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.335989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3043ba-32b5-4c15-bb95-4569c88fff70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsz5\" (UniqueName: \"kubernetes.io/projected/fc3043ba-32b5-4c15-bb95-4569c88fff70-kube-api-access-kxsz5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336148 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336432 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336432 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.336509 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.336477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.338594 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.338569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3043ba-32b5-4c15-bb95-4569c88fff70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.338737 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.338666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3043ba-32b5-4c15-bb95-4569c88fff70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.343384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.343358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsz5\" (UniqueName: \"kubernetes.io/projected/fc3043ba-32b5-4c15-bb95-4569c88fff70-kube-api-access-kxsz5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v\" (UID: \"fc3043ba-32b5-4c15-bb95-4569c88fff70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.493901 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.493812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:38.656500 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:38.656110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"]
Jul 29 21:16:39.533712 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:39.533654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v" event={"ID":"fc3043ba-32b5-4c15-bb95-4569c88fff70","Type":"ContainerStarted","Data":"5ebe30a4b73d6b5710cd35a2bacf762d2510e74b23ca9e48e42ce5c5769675c8"}
Jul 29 21:16:40.540295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:40.540254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v" event={"ID":"fc3043ba-32b5-4c15-bb95-4569c88fff70","Type":"ContainerStarted","Data":"183426e0ca0ef084b13df2822bb3273741abdaf086f9abc262ba31a7305992da"}
Jul 29 21:16:40.558476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:40.558411    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v" podStartSLOduration=1.308838983 podStartE2EDuration="2.558391226s" totalImagesPullingTime="1.249552243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:40.55611373 +0000 UTC m=+787.854591487" watchObservedRunningTime="2026-07-29 21:16:40.558391226 +0000 UTC m=+787.856868981"
Jul 29 21:16:48.494606 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:48.494568    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:48.495088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:48.494624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:48.506993 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:48.506965    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:16:48.592861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:16:48.592832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v"
Jul 29 21:17:02.283755 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.283647    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"]
Jul 29 21:17:02.287368 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.287344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.289886 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.289860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 21:17:02.296385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.296356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"]
Jul 29 21:17:02.352417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.352417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qrb\" (UniqueName: \"kubernetes.io/projected/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-kube-api-access-s7qrb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.352645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.352645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.352645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.352645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.352576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453415 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qrb\" (UniqueName: \"kubernetes.io/projected/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-kube-api-access-s7qrb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.453687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.453652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.454138 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.454065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.456036 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.455997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.456188 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.456112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.465870 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.465829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qrb\" (UniqueName: \"kubernetes.io/projected/e8f01d9a-d3ee-4278-915f-3f4d6e335d37-kube-api-access-s7qrb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg\" (UID: \"e8f01d9a-d3ee-4278-915f-3f4d6e335d37\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.600228 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.600194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:02.775154 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:02.775122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"]
Jul 29 21:17:02.777092 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:17:02.777058    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f01d9a_d3ee_4278_915f_3f4d6e335d37.slice/crio-19a737bea441e34b95e6874cd98ea895ec985449998a694fd39f50eae570d59a WatchSource:0}: Error finding container 19a737bea441e34b95e6874cd98ea895ec985449998a694fd39f50eae570d59a: Status 404 returned error can't find the container with id 19a737bea441e34b95e6874cd98ea895ec985449998a694fd39f50eae570d59a
Jul 29 21:17:03.667067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:03.660425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg" event={"ID":"e8f01d9a-d3ee-4278-915f-3f4d6e335d37","Type":"ContainerStarted","Data":"5ecedf37b094cc385074aaf5f9cb60bc9f934936bc6b09faf1fa859a970b2017"}
Jul 29 21:17:03.667067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:03.660912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg" event={"ID":"e8f01d9a-d3ee-4278-915f-3f4d6e335d37","Type":"ContainerStarted","Data":"19a737bea441e34b95e6874cd98ea895ec985449998a694fd39f50eae570d59a"}
Jul 29 21:17:03.682188 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:03.682126    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg" podStartSLOduration=1.486212693 podStartE2EDuration="1.682108815s" totalImagesPullingTime="195.896122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:03.679572485 +0000 UTC m=+810.978050240" watchObservedRunningTime="2026-07-29 21:17:03.682108815 +0000 UTC m=+810.980586570"
Jul 29 21:17:10.785940 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.785900    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"]
Jul 29 21:17:10.789610 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.789585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.791897 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.791872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 29 21:17:10.800135 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.799938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"]
Jul 29 21:17:10.821525 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0dbec1-f992-491e-b6fd-2795caebbeea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.821683 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.821683 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrtn\" (UniqueName: \"kubernetes.io/projected/6f0dbec1-f992-491e-b6fd-2795caebbeea-kube-api-access-qbrtn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.821683 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.821683 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.821854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.821722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrtn\" (UniqueName: \"kubernetes.io/projected/6f0dbec1-f992-491e-b6fd-2795caebbeea-kube-api-access-qbrtn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0dbec1-f992-491e-b6fd-2795caebbeea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922673 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.922768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.922690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.924906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.924875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0dbec1-f992-491e-b6fd-2795caebbeea-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.925109 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.925090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0dbec1-f992-491e-b6fd-2795caebbeea-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:10.930068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:10.930046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrtn\" (UniqueName: \"kubernetes.io/projected/6f0dbec1-f992-491e-b6fd-2795caebbeea-kube-api-access-qbrtn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g2d4q\" (UID: \"6f0dbec1-f992-491e-b6fd-2795caebbeea\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:11.104214 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:11.104171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:11.254917 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:11.254883    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"]
Jul 29 21:17:11.258453 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:17:11.258423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0dbec1_f992_491e_b6fd_2795caebbeea.slice/crio-8a4f662b7f7ae213f2b760d369d9cde2f2467bfb5e288e40de2a5fe3a09c4c36 WatchSource:0}: Error finding container 8a4f662b7f7ae213f2b760d369d9cde2f2467bfb5e288e40de2a5fe3a09c4c36: Status 404 returned error can't find the container with id 8a4f662b7f7ae213f2b760d369d9cde2f2467bfb5e288e40de2a5fe3a09c4c36
Jul 29 21:17:11.707266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:11.707160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q" event={"ID":"6f0dbec1-f992-491e-b6fd-2795caebbeea","Type":"ContainerStarted","Data":"66a4c7f46705983b25e6bbff1913ce80bc22e75d3642af0ac4f6f6083b5cb26c"}
Jul 29 21:17:11.707266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:11.707233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q" event={"ID":"6f0dbec1-f992-491e-b6fd-2795caebbeea","Type":"ContainerStarted","Data":"8a4f662b7f7ae213f2b760d369d9cde2f2467bfb5e288e40de2a5fe3a09c4c36"}
Jul 29 21:17:11.727444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:11.727385    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q" podStartSLOduration=1.550900329 podStartE2EDuration="1.727364649s" totalImagesPullingTime="176.46432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:11.72584825 +0000 UTC m=+819.024326002" watchObservedRunningTime="2026-07-29 21:17:11.727364649 +0000 UTC m=+819.025842456"
Jul 29 21:17:12.600866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:12.600825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:12.601464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:12.600904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:12.613688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:12.613656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:12.725118 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:12.725088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg"
Jul 29 21:17:19.291084 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.291038    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"]
Jul 29 21:17:19.295388 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.295360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.297730 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.297681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 21:17:19.303487 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.303458    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"]
Jul 29 21:17:19.384780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.384740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.384970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.384789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gx8b\" (UniqueName: \"kubernetes.io/projected/694f2fcd-0c10-4847-804b-c0d880901186-kube-api-access-9gx8b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.384970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.384838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.384970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.384899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694f2fcd-0c10-4847-804b-c0d880901186-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.384970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.384928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.385118 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.385004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694f2fcd-0c10-4847-804b-c0d880901186-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gx8b\" (UniqueName: \"kubernetes.io/projected/694f2fcd-0c10-4847-804b-c0d880901186-kube-api-access-9gx8b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.485986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.485972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.486284 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.486262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.486352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.486296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.486352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.486340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.488213 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.488189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694f2fcd-0c10-4847-804b-c0d880901186-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.488463 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.488446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694f2fcd-0c10-4847-804b-c0d880901186-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.495834 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.495812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gx8b\" (UniqueName: \"kubernetes.io/projected/694f2fcd-0c10-4847-804b-c0d880901186-kube-api-access-9gx8b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-58thp\" (UID: \"694f2fcd-0c10-4847-804b-c0d880901186\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.607563 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.607522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:19.759407 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:19.759367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"]
Jul 29 21:17:19.760268 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:17:19.760218    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod694f2fcd_0c10_4847_804b_c0d880901186.slice/crio-da792708f5d8c80a9b361e9ffbc96876ee4bcbd916fc2b57412d43dc11fcbdf6 WatchSource:0}: Error finding container da792708f5d8c80a9b361e9ffbc96876ee4bcbd916fc2b57412d43dc11fcbdf6: Status 404 returned error can't find the container with id da792708f5d8c80a9b361e9ffbc96876ee4bcbd916fc2b57412d43dc11fcbdf6
Jul 29 21:17:20.760432 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:20.760395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp" event={"ID":"694f2fcd-0c10-4847-804b-c0d880901186","Type":"ContainerStarted","Data":"e0d1cffc908eea9a81e349a339107f7f7731e385d1f8afd2df038dcbe4e8be04"}
Jul 29 21:17:20.760432 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:20.760439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp" event={"ID":"694f2fcd-0c10-4847-804b-c0d880901186","Type":"ContainerStarted","Data":"da792708f5d8c80a9b361e9ffbc96876ee4bcbd916fc2b57412d43dc11fcbdf6"}
Jul 29 21:17:20.778017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:20.777961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp" podStartSLOduration=1.574847173 podStartE2EDuration="1.777943713s" totalImagesPullingTime="203.09654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:20.775805923 +0000 UTC m=+828.074283690" watchObservedRunningTime="2026-07-29 21:17:20.777943713 +0000 UTC m=+828.076421472"
Jul 29 21:17:21.104799 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:21.104762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:21.104994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:21.104837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:21.118135 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:21.118102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:21.778768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:21.778732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g2d4q"
Jul 29 21:17:29.608151 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:29.608100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:29.608559 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:29.608168    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:29.621116 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:29.621080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:17:29.822781 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:17:29.822748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-58thp"
Jul 29 21:18:27.881563 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:27.881454    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c986c7ccc-w2cxs"]
Jul 29 21:18:27.885196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:27.885172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:27.892482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:27.892448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c986c7ccc-w2cxs"]
Jul 29 21:18:27.997856 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:27.997823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b12b766-ac8c-489f-8910-acaedac3a00e-tls-cert\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:27.998052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:27.997866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s2j\" (UniqueName: \"kubernetes.io/projected/6b12b766-ac8c-489f-8910-acaedac3a00e-kube-api-access-n4s2j\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.099179 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.099142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b12b766-ac8c-489f-8910-acaedac3a00e-tls-cert\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.099385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.099187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s2j\" (UniqueName: \"kubernetes.io/projected/6b12b766-ac8c-489f-8910-acaedac3a00e-kube-api-access-n4s2j\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.101910 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.101886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b12b766-ac8c-489f-8910-acaedac3a00e-tls-cert\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.108242 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.108214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s2j\" (UniqueName: \"kubernetes.io/projected/6b12b766-ac8c-489f-8910-acaedac3a00e-kube-api-access-n4s2j\") pod \"authorino-5c986c7ccc-w2cxs\" (UID: \"6b12b766-ac8c-489f-8910-acaedac3a00e\") " pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.216606 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.216520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c986c7ccc-w2cxs"
Jul 29 21:18:28.369170 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:28.369141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c986c7ccc-w2cxs"]
Jul 29 21:18:28.370745 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:18:28.370712    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b12b766_ac8c_489f_8910_acaedac3a00e.slice/crio-6cfac061eb097cb6d294b2800fc770844a9901f124a32cb129a5a89df43417f0 WatchSource:0}: Error finding container 6cfac061eb097cb6d294b2800fc770844a9901f124a32cb129a5a89df43417f0: Status 404 returned error can't find the container with id 6cfac061eb097cb6d294b2800fc770844a9901f124a32cb129a5a89df43417f0
Jul 29 21:18:29.135928 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.135888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c986c7ccc-w2cxs" event={"ID":"6b12b766-ac8c-489f-8910-acaedac3a00e","Type":"ContainerStarted","Data":"61342ac73d07f63b410c9034d845fd9c437589df75d2ff14266ccf9aba287f4d"}
Jul 29 21:18:29.136320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.135944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c986c7ccc-w2cxs" event={"ID":"6b12b766-ac8c-489f-8910-acaedac3a00e","Type":"ContainerStarted","Data":"6cfac061eb097cb6d294b2800fc770844a9901f124a32cb129a5a89df43417f0"}
Jul 29 21:18:29.156348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.156256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c986c7ccc-w2cxs" podStartSLOduration=1.766076284 podStartE2EDuration="2.156231582s" totalImagesPullingTime="390.155298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:29.153923143 +0000 UTC m=+896.452400898" watchObservedRunningTime="2026-07-29 21:18:29.156231582 +0000 UTC m=+896.454709338"
Jul 29 21:18:29.194937 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.194479    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:18:29.194937 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.194620    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bd49d8576-bdn7m" podUID="1c491116-442a-48f1-895b-e7a567880aae" containerName="authorino" containerID="cri-o://bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea" gracePeriod=30
Jul 29 21:18:29.465823 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.465789    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:18:29.514008 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.513967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8jwc\" (UniqueName: \"kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc\") pod \"1c491116-442a-48f1-895b-e7a567880aae\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") "
Jul 29 21:18:29.514207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.514023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert\") pod \"1c491116-442a-48f1-895b-e7a567880aae\" (UID: \"1c491116-442a-48f1-895b-e7a567880aae\") "
Jul 29 21:18:29.516249 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.516214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc" (OuterVolumeSpecName: "kube-api-access-t8jwc") pod "1c491116-442a-48f1-895b-e7a567880aae" (UID: "1c491116-442a-48f1-895b-e7a567880aae"). InnerVolumeSpecName "kube-api-access-t8jwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:29.525644 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.525603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1c491116-442a-48f1-895b-e7a567880aae" (UID: "1c491116-442a-48f1-895b-e7a567880aae"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:29.615754 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.615689    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8jwc\" (UniqueName: \"kubernetes.io/projected/1c491116-442a-48f1-895b-e7a567880aae-kube-api-access-t8jwc\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.615754 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:29.615758    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c491116-442a-48f1-895b-e7a567880aae-tls-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:18:30.144764 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.144731    2581 generic.go:356] "Generic (PLEG): container finished" podID="1c491116-442a-48f1-895b-e7a567880aae" containerID="bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea" exitCode=0
Jul 29 21:18:30.145206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.144790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd49d8576-bdn7m"
Jul 29 21:18:30.145206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.144825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd49d8576-bdn7m" event={"ID":"1c491116-442a-48f1-895b-e7a567880aae","Type":"ContainerDied","Data":"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"}
Jul 29 21:18:30.145206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.144879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd49d8576-bdn7m" event={"ID":"1c491116-442a-48f1-895b-e7a567880aae","Type":"ContainerDied","Data":"2a12104fc011a78cbc3db2c64a2d4b032a36b5e07797ed8471e7fdf98f41e711"}
Jul 29 21:18:30.145206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.144901    2581 scope.go:117] "RemoveContainer" containerID="bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"
Jul 29 21:18:30.159046 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.159021    2581 scope.go:117] "RemoveContainer" containerID="bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"
Jul 29 21:18:30.159727 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:18:30.159673    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea\": container with ID starting with bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea not found: ID does not exist" containerID="bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"
Jul 29 21:18:30.159836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.159743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea"} err="failed to get container status \"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea\": rpc error: code = NotFound desc = could not find container \"bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea\": container with ID starting with bd249baca5d1498f01d03377f516ad43227ebc3291eff3cf7333c269320197ea not found: ID does not exist"
Jul 29 21:18:30.198592 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.198554    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:18:30.211017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:30.210978    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bd49d8576-bdn7m"]
Jul 29 21:18:31.187899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:31.187867    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c491116-442a-48f1-895b-e7a567880aae" path="/var/lib/kubelet/pods/1c491116-442a-48f1-895b-e7a567880aae/volumes"
Jul 29 21:18:33.111407 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:33.111377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:18:33.112240 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:18:33.112216    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:19:17.646911 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:17.646858    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:19:17.647499 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:17.647113    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" podUID="040ecb7d-0f30-410f-b623-690dfa00ecb2" containerName="manager" containerID="cri-o://530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b" gracePeriod=30
Jul 29 21:19:17.914930 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:17.914901    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:19:18.044566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.044513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqc85\" (UniqueName: \"kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85\") pod \"040ecb7d-0f30-410f-b623-690dfa00ecb2\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") "
Jul 29 21:19:18.044794 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.044586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert\") pod \"040ecb7d-0f30-410f-b623-690dfa00ecb2\" (UID: \"040ecb7d-0f30-410f-b623-690dfa00ecb2\") "
Jul 29 21:19:18.046962 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.046921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85" (OuterVolumeSpecName: "kube-api-access-xqc85") pod "040ecb7d-0f30-410f-b623-690dfa00ecb2" (UID: "040ecb7d-0f30-410f-b623-690dfa00ecb2"). InnerVolumeSpecName "kube-api-access-xqc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:19:18.046962 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.046947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "040ecb7d-0f30-410f-b623-690dfa00ecb2" (UID: "040ecb7d-0f30-410f-b623-690dfa00ecb2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.145183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.145145    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqc85\" (UniqueName: \"kubernetes.io/projected/040ecb7d-0f30-410f-b623-690dfa00ecb2-kube-api-access-xqc85\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.145183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.145176    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/040ecb7d-0f30-410f-b623-690dfa00ecb2-webhook-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.417017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.416986    2581 generic.go:356] "Generic (PLEG): container finished" podID="040ecb7d-0f30-410f-b623-690dfa00ecb2" containerID="530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b" exitCode=0
Jul 29 21:19:18.417218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.417050    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-6hr9c"
Jul 29 21:19:18.417218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.417070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" event={"ID":"040ecb7d-0f30-410f-b623-690dfa00ecb2","Type":"ContainerDied","Data":"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"}
Jul 29 21:19:18.417218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.417125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-6hr9c" event={"ID":"040ecb7d-0f30-410f-b623-690dfa00ecb2","Type":"ContainerDied","Data":"9a99e2c415745847226d9e288f2d4a5dd3e997fa0c1f06636a6c6c35f7572997"}
Jul 29 21:19:18.417218 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.417139    2581 scope.go:117] "RemoveContainer" containerID="530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"
Jul 29 21:19:18.427034 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.427016    2581 scope.go:117] "RemoveContainer" containerID="530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"
Jul 29 21:19:18.427345 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:19:18.427322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b\": container with ID starting with 530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b not found: ID does not exist" containerID="530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"
Jul 29 21:19:18.427397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.427354    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b"} err="failed to get container status \"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b\": rpc error: code = NotFound desc = could not find container \"530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b\": container with ID starting with 530157536caa88495d147c0f500b8ecfa31dc11aa5e4a2059640abf0d0ae3b5b not found: ID does not exist"
Jul 29 21:19:18.439208 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.439178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:19:18.443436 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:18.443405    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-6hr9c"]
Jul 29 21:19:19.187727 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:19:19.187677    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="040ecb7d-0f30-410f-b623-690dfa00ecb2" path="/var/lib/kubelet/pods/040ecb7d-0f30-410f-b623-690dfa00ecb2/volumes"
Jul 29 21:20:19.344552 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344513    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040ecb7d-0f30-410f-b623-690dfa00ecb2" containerName="manager"
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344907    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="040ecb7d-0f30-410f-b623-690dfa00ecb2" containerName="manager"
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344932    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c491116-442a-48f1-895b-e7a567880aae" containerName="authorino"
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344937    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c491116-442a-48f1-895b-e7a567880aae" containerName="authorino"
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.344992    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="040ecb7d-0f30-410f-b623-690dfa00ecb2" containerName="manager"
Jul 29 21:20:19.345043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.345002    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c491116-442a-48f1-895b-e7a567880aae" containerName="authorino"
Jul 29 21:20:19.348125 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.348109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.351798 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.351766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:20:19.351798 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.351785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:20:19.351991 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.351785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-slf4h\""
Jul 29 21:20:19.357049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.357021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:19.474203 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.474157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbpw\" (UniqueName: \"kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.474398 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.474267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.575769 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.575728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbpw\" (UniqueName: \"kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.575913 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.575813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.578464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.578437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.585913 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.585892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbpw\" (UniqueName: \"kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw\") pod \"maas-controller-6469f67b9b-ps2xt\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") " pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.660977 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.660940    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:19.807583 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.807555    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:19.809661 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:20:19.809631    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67e474d_b067_4061_b41b_1096f975bc09.slice/crio-27ecf8238cb5a04f165cda0214b6762e0b242c509c1705950e86b94c24ccfb43 WatchSource:0}: Error finding container 27ecf8238cb5a04f165cda0214b6762e0b242c509c1705950e86b94c24ccfb43: Status 404 returned error can't find the container with id 27ecf8238cb5a04f165cda0214b6762e0b242c509c1705950e86b94c24ccfb43
Jul 29 21:20:19.811507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:19.811485    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:20:20.767295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:20.767189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" event={"ID":"d67e474d-b067-4061-b41b-1096f975bc09","Type":"ContainerStarted","Data":"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"}
Jul 29 21:20:20.767295 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:20.767257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" event={"ID":"d67e474d-b067-4061-b41b-1096f975bc09","Type":"ContainerStarted","Data":"27ecf8238cb5a04f165cda0214b6762e0b242c509c1705950e86b94c24ccfb43"}
Jul 29 21:20:20.767830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:20.767297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:20.788176 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:20.788117    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" podStartSLOduration=1.198820277 podStartE2EDuration="1.78809558s" totalImagesPullingTime="589.275303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:20.786173725 +0000 UTC m=+1008.084651480" watchObservedRunningTime="2026-07-29 21:20:20.78809558 +0000 UTC m=+1008.086573335"
Jul 29 21:20:31.775393 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:31.775356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:36.477368 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.477324    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:36.477859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.477570    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" podUID="d67e474d-b067-4061-b41b-1096f975bc09" containerName="manager" containerID="cri-o://c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529" gracePeriod=30
Jul 29 21:20:36.737989 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.737919    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:36.831024 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.830988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert\") pod \"d67e474d-b067-4061-b41b-1096f975bc09\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") "
Jul 29 21:20:36.831200 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.831044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tbpw\" (UniqueName: \"kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw\") pod \"d67e474d-b067-4061-b41b-1096f975bc09\" (UID: \"d67e474d-b067-4061-b41b-1096f975bc09\") "
Jul 29 21:20:36.833231 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.833196    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d67e474d-b067-4061-b41b-1096f975bc09" (UID: "d67e474d-b067-4061-b41b-1096f975bc09"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:20:36.833417 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.833391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw" (OuterVolumeSpecName: "kube-api-access-8tbpw") pod "d67e474d-b067-4061-b41b-1096f975bc09" (UID: "d67e474d-b067-4061-b41b-1096f975bc09"). InnerVolumeSpecName "kube-api-access-8tbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:20:36.853239 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.853207    2581 generic.go:356] "Generic (PLEG): container finished" podID="d67e474d-b067-4061-b41b-1096f975bc09" containerID="c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529" exitCode=0
Jul 29 21:20:36.853434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.853249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-ps2xt"
Jul 29 21:20:36.853434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.853293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" event={"ID":"d67e474d-b067-4061-b41b-1096f975bc09","Type":"ContainerDied","Data":"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"}
Jul 29 21:20:36.853434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.853355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-ps2xt" event={"ID":"d67e474d-b067-4061-b41b-1096f975bc09","Type":"ContainerDied","Data":"27ecf8238cb5a04f165cda0214b6762e0b242c509c1705950e86b94c24ccfb43"}
Jul 29 21:20:36.853434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.853386    2581 scope.go:117] "RemoveContainer" containerID="c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"
Jul 29 21:20:36.865074 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.865039    2581 scope.go:117] "RemoveContainer" containerID="c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"
Jul 29 21:20:36.865400 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:20:36.865380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529\": container with ID starting with c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529 not found: ID does not exist" containerID="c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"
Jul 29 21:20:36.865464 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.865409    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529"} err="failed to get container status \"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529\": rpc error: code = NotFound desc = could not find container \"c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529\": container with ID starting with c3f9a92532bc2935a00b04a2267ac0e4e61210ffea753977dc61feb971917529 not found: ID does not exist"
Jul 29 21:20:36.876323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.876288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:36.880597 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.880567    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-ps2xt"]
Jul 29 21:20:36.932466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.932426    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67e474d-b067-4061-b41b-1096f975bc09-webhook-cert\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:20:36.932466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:36.932460    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbpw\" (UniqueName: \"kubernetes.io/projected/d67e474d-b067-4061-b41b-1096f975bc09-kube-api-access-8tbpw\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:20:37.187902 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.187866    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67e474d-b067-4061-b41b-1096f975bc09" path="/var/lib/kubelet/pods/d67e474d-b067-4061-b41b-1096f975bc09/volumes"
Jul 29 21:20:37.714453 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.714417    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6469f67b9b-bw85b"]
Jul 29 21:20:37.714876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.714830    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67e474d-b067-4061-b41b-1096f975bc09" containerName="manager"
Jul 29 21:20:37.714876 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.714843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67e474d-b067-4061-b41b-1096f975bc09" containerName="manager"
Jul 29 21:20:37.714949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.714901    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67e474d-b067-4061-b41b-1096f975bc09" containerName="manager"
Jul 29 21:20:37.719302 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.719283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.723004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.722969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-slf4h\""
Jul 29 21:20:37.723004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.722984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:20:37.723004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.722984    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:20:37.732364 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.732335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-bw85b"]
Jul 29 21:20:37.840866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.840823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb72052e-be0a-453e-b486-393662938058-webhook-cert\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.841052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.840881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfg9\" (UniqueName: \"kubernetes.io/projected/eb72052e-be0a-453e-b486-393662938058-kube-api-access-5dfg9\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.942323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.942274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb72052e-be0a-453e-b486-393662938058-webhook-cert\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.942323 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.942321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfg9\" (UniqueName: \"kubernetes.io/projected/eb72052e-be0a-453e-b486-393662938058-kube-api-access-5dfg9\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.945075 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.945048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb72052e-be0a-453e-b486-393662938058-webhook-cert\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:37.952628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:37.952604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfg9\" (UniqueName: \"kubernetes.io/projected/eb72052e-be0a-453e-b486-393662938058-kube-api-access-5dfg9\") pod \"maas-controller-6469f67b9b-bw85b\" (UID: \"eb72052e-be0a-453e-b486-393662938058\") " pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:38.030416 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:38.030325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:38.180203 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:38.180178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6469f67b9b-bw85b"]
Jul 29 21:20:38.182241 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:20:38.182208    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb72052e_be0a_453e_b486_393662938058.slice/crio-67f649d805677111d93f5803a904d1def2de1897c6f215676837fa90572fae3c WatchSource:0}: Error finding container 67f649d805677111d93f5803a904d1def2de1897c6f215676837fa90572fae3c: Status 404 returned error can't find the container with id 67f649d805677111d93f5803a904d1def2de1897c6f215676837fa90572fae3c
Jul 29 21:20:38.866770 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:38.866728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-bw85b" event={"ID":"eb72052e-be0a-453e-b486-393662938058","Type":"ContainerStarted","Data":"67f649d805677111d93f5803a904d1def2de1897c6f215676837fa90572fae3c"}
Jul 29 21:20:39.873431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:39.873393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6469f67b9b-bw85b" event={"ID":"eb72052e-be0a-453e-b486-393662938058","Type":"ContainerStarted","Data":"d56fa90ab8d2593b5eed41f97422883ec2c4aa746ab1dcc6f4170ac098990436"}
Jul 29 21:20:39.873872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:39.873551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:20:39.892527 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:39.892457    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6469f67b9b-bw85b" podStartSLOduration=2.157712501 podStartE2EDuration="2.89243892s" totalImagesPullingTime="734.726419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:39.891176499 +0000 UTC m=+1027.189654257" watchObservedRunningTime="2026-07-29 21:20:39.89243892 +0000 UTC m=+1027.190916676"
Jul 29 21:20:50.881996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:20:50.881963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6469f67b9b-bw85b"
Jul 29 21:23:33.156631 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:23:33.156602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:23:33.161773 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:23:33.161741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:25:47.088331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:47.088290    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:25:47.091101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:47.088620    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" podUID="5e59880b-d47f-4961-a4bd-931a22415090" containerName="manager" containerID="cri-o://eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01" gracePeriod=10
Jul 29 21:25:50.441786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.441752    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:25:50.542246 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.542141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtm98\" (UniqueName: \"kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98\") pod \"5e59880b-d47f-4961-a4bd-931a22415090\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") "
Jul 29 21:25:50.542246 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.542243    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume\") pod \"5e59880b-d47f-4961-a4bd-931a22415090\" (UID: \"5e59880b-d47f-4961-a4bd-931a22415090\") "
Jul 29 21:25:50.542625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.542600    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5e59880b-d47f-4961-a4bd-931a22415090" (UID: "5e59880b-d47f-4961-a4bd-931a22415090"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:50.544431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.544406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98" (OuterVolumeSpecName: "kube-api-access-jtm98") pod "5e59880b-d47f-4961-a4bd-931a22415090" (UID: "5e59880b-d47f-4961-a4bd-931a22415090"). InnerVolumeSpecName "kube-api-access-jtm98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:25:50.560342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.560313    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e59880b-d47f-4961-a4bd-931a22415090" containerID="eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01" exitCode=0
Jul 29 21:25:50.560482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.560358    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"
Jul 29 21:25:50.560482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.560394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" event={"ID":"5e59880b-d47f-4961-a4bd-931a22415090","Type":"ContainerDied","Data":"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"}
Jul 29 21:25:50.560482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.560451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96" event={"ID":"5e59880b-d47f-4961-a4bd-931a22415090","Type":"ContainerDied","Data":"ab5e6a1ad270169e46f6e489491a1e5d5a7455ac0f2587fe3c42fc20cec308a4"}
Jul 29 21:25:50.560482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.560474    2581 scope.go:117] "RemoveContainer" containerID="eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"
Jul 29 21:25:50.570982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.570958    2581 scope.go:117] "RemoveContainer" containerID="eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"
Jul 29 21:25:50.571301 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:25:50.571281    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01\": container with ID starting with eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01 not found: ID does not exist" containerID="eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"
Jul 29 21:25:50.571371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.571310    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01"} err="failed to get container status \"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01\": rpc error: code = NotFound desc = could not find container \"eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01\": container with ID starting with eae02aff991718c3658f60adf257a1c80576c5df0f816e159b704b1b5e54bc01 not found: ID does not exist"
Jul 29 21:25:50.583823 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.583792    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:25:50.588588 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.588559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2nh96"]
Jul 29 21:25:50.642804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.642760    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtm98\" (UniqueName: \"kubernetes.io/projected/5e59880b-d47f-4961-a4bd-931a22415090-kube-api-access-jtm98\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:25:50.642804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:50.642800    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e59880b-d47f-4961-a4bd-931a22415090-extensions-socket-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:25:51.188269 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:25:51.188236    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e59880b-d47f-4961-a4bd-931a22415090" path="/var/lib/kubelet/pods/5e59880b-d47f-4961-a4bd-931a22415090/volumes"
Jul 29 21:26:48.312614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.312577    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"]
Jul 29 21:26:48.313040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.312938    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e59880b-d47f-4961-a4bd-931a22415090" containerName="manager"
Jul 29 21:26:48.313040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.312949    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e59880b-d47f-4961-a4bd-931a22415090" containerName="manager"
Jul 29 21:26:48.313040 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.313016    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e59880b-d47f-4961-a4bd-931a22415090" containerName="manager"
Jul 29 21:26:48.317186 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.317167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.319861 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.319839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4hrql\""
Jul 29 21:26:48.328563 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.328540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"]
Jul 29 21:26:48.458977 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.458944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8n7s\" (UniqueName: \"kubernetes.io/projected/82559be4-d305-4ac1-93a8-fe85fc07a4ca-kube-api-access-p8n7s\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.459167 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.459008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82559be4-d305-4ac1-93a8-fe85fc07a4ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.560275 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.560232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82559be4-d305-4ac1-93a8-fe85fc07a4ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.560446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.560320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8n7s\" (UniqueName: \"kubernetes.io/projected/82559be4-d305-4ac1-93a8-fe85fc07a4ca-kube-api-access-p8n7s\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.560634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.560613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82559be4-d305-4ac1-93a8-fe85fc07a4ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.570403 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.570345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8n7s\" (UniqueName: \"kubernetes.io/projected/82559be4-d305-4ac1-93a8-fe85fc07a4ca-kube-api-access-p8n7s\") pod \"kuadrant-operator-controller-manager-86878c769f-jsz76\" (UID: \"82559be4-d305-4ac1-93a8-fe85fc07a4ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.628114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.628078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.786610 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.786573    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"]
Jul 29 21:26:48.788009 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:26:48.787975    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82559be4_d305_4ac1_93a8_fe85fc07a4ca.slice/crio-03a9e28c66070e664db03a623d723bc4a29f90d9aee62d1d5d236690e94f16f0 WatchSource:0}: Error finding container 03a9e28c66070e664db03a623d723bc4a29f90d9aee62d1d5d236690e94f16f0: Status 404 returned error can't find the container with id 03a9e28c66070e664db03a623d723bc4a29f90d9aee62d1d5d236690e94f16f0
Jul 29 21:26:48.877200 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.877166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76" event={"ID":"82559be4-d305-4ac1-93a8-fe85fc07a4ca","Type":"ContainerStarted","Data":"62251f163dc8b1fbf758c7ee191f13ee818da863a536bfb99be813eeeafdf04d"}
Jul 29 21:26:48.877401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.877219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76" event={"ID":"82559be4-d305-4ac1-93a8-fe85fc07a4ca","Type":"ContainerStarted","Data":"03a9e28c66070e664db03a623d723bc4a29f90d9aee62d1d5d236690e94f16f0"}
Jul 29 21:26:48.877401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.877275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:26:48.901340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:48.901278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76" podStartSLOduration=0.901262375 podStartE2EDuration="901.262375ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:48.898838671 +0000 UTC m=+1396.197316426" watchObservedRunningTime="2026-07-29 21:26:48.901262375 +0000 UTC m=+1396.199740130"
Jul 29 21:26:59.884986 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:26:59.884952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jsz76"
Jul 29 21:28:33.191576 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:28:33.191546    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:28:33.197789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:28:33.197767    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:33:33.226718 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:33:33.226676    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:33:33.234339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:33:33.234314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:34:16.458048 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.458013    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:16.461481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.461456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.464398 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.464376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-80781434-serving-cert\""
Jul 29 21:34:16.472569 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.472538    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:16.546285 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.546245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.546285 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.546286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24jt\" (UniqueName: \"kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.546522 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.546330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.647422 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.647389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.647622 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.647429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24jt\" (UniqueName: \"kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.647622 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:16.647543    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-80781434-serving-cert: secret "maas-api-e2e-ait-80781434-serving-cert" not found
Jul 29 21:34:16.647622 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:16.647613    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls podName:45907e52-9928-47b3-820c-79b4fcf96638 nodeName:}" failed. No retries permitted until 2026-07-29 21:34:17.147587301 +0000 UTC m=+1844.446065034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls") pod "maas-api-e2e-ait-80781434-68996f7f88-mtwj4" (UID: "45907e52-9928-47b3-820c-79b4fcf96638") : secret "maas-api-e2e-ait-80781434-serving-cert" not found
Jul 29 21:34:16.647849 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.647634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.648363 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.648337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:16.656297 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:16.656270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24jt\" (UniqueName: \"kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:17.153071 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.153012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:17.155919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.155887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") pod \"maas-api-e2e-ait-80781434-68996f7f88-mtwj4\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:17.249555 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.249516    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:17.253294 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.253270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.255945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.255917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-80781434-dockercfg-4cfh2\""
Jul 29 21:34:17.256116 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.256000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-80781434\""
Jul 29 21:34:17.264807 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.264773    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:17.317353 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.317319    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:17.320748 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.320721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.329202 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.329175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:17.355253 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.355137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7fc\" (UniqueName: \"kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.355461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.355297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.355461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.355393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqvq\" (UniqueName: \"kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.355461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.355421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.374314 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.374275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:17.456534 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.456497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqvq\" (UniqueName: \"kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.456726 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.456542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.456726 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.456589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7fc\" (UniqueName: \"kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.456726 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.456625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.457510 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.457486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.457675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.457657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.468902 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.468873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7fc\" (UniqueName: \"kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc\") pod \"payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") " pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.469240 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.468940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqvq\" (UniqueName: \"kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq\") pod \"payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.535377 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.535343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:17.537155 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:34:17.537122    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45907e52_9928_47b3_820c_79b4fcf96638.slice/crio-f9d4d6843e4499fa9129df92f1069e7b58251bff938505249c8664b9e4d4d1f0 WatchSource:0}: Error finding container f9d4d6843e4499fa9129df92f1069e7b58251bff938505249c8664b9e4d4d1f0: Status 404 returned error can't find the container with id f9d4d6843e4499fa9129df92f1069e7b58251bff938505249c8664b9e4d4d1f0
Jul 29 21:34:17.538386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.538369    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:34:17.572752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.572714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:17.633289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.633251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:17.732054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.732024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:17.733827 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:34:17.733794    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1260a5fa_66a4_4a9c_9714_1f1ec4d7a06f.slice/crio-e167dd753d021ee2ff3bcc4992d63f2c18017aed8ef910d938d104588224f804 WatchSource:0}: Error finding container e167dd753d021ee2ff3bcc4992d63f2c18017aed8ef910d938d104588224f804: Status 404 returned error can't find the container with id e167dd753d021ee2ff3bcc4992d63f2c18017aed8ef910d938d104588224f804
Jul 29 21:34:17.800359 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:17.800332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:17.802750 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:34:17.802717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6019a732_1109_4dfa_bf39_8a9be727e226.slice/crio-2c50686c4a13da5fa3c46b68fcc3391a4ba550dbb62b06997c9eff4eb9371ba2 WatchSource:0}: Error finding container 2c50686c4a13da5fa3c46b68fcc3391a4ba550dbb62b06997c9eff4eb9371ba2: Status 404 returned error can't find the container with id 2c50686c4a13da5fa3c46b68fcc3391a4ba550dbb62b06997c9eff4eb9371ba2
Jul 29 21:34:18.375864 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.375744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" event={"ID":"6019a732-1109-4dfa-bf39-8a9be727e226","Type":"ContainerStarted","Data":"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"}
Jul 29 21:34:18.375864 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.375821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" event={"ID":"6019a732-1109-4dfa-bf39-8a9be727e226","Type":"ContainerStarted","Data":"2c50686c4a13da5fa3c46b68fcc3391a4ba550dbb62b06997c9eff4eb9371ba2"}
Jul 29 21:34:18.376118 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.376009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:18.377891 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.377861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" event={"ID":"45907e52-9928-47b3-820c-79b4fcf96638","Type":"ContainerStarted","Data":"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"}
Jul 29 21:34:18.378067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.377935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" event={"ID":"45907e52-9928-47b3-820c-79b4fcf96638","Type":"ContainerStarted","Data":"f9d4d6843e4499fa9129df92f1069e7b58251bff938505249c8664b9e4d4d1f0"}
Jul 29 21:34:18.378067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.377969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:18.379595 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.379569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" event={"ID":"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f","Type":"ContainerStarted","Data":"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"}
Jul 29 21:34:18.379691 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.379617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" event={"ID":"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f","Type":"ContainerStarted","Data":"e167dd753d021ee2ff3bcc4992d63f2c18017aed8ef910d938d104588224f804"}
Jul 29 21:34:18.379691 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.379645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:18.411872 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.411812    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" podStartSLOduration=1.411793002 podStartE2EDuration="1.411793002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:18.393249479 +0000 UTC m=+1845.691727235" watchObservedRunningTime="2026-07-29 21:34:18.411793002 +0000 UTC m=+1845.710270757"
Jul 29 21:34:18.413017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.412986    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" podStartSLOduration=1.849486916 podStartE2EDuration="2.412972375s" totalImagesPullingTime="563.485459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:18.411105511 +0000 UTC m=+1845.709583268" watchObservedRunningTime="2026-07-29 21:34:18.412972375 +0000 UTC m=+1845.711450182"
Jul 29 21:34:18.428624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:18.428565    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" podStartSLOduration=1.4285492419999999 podStartE2EDuration="1.428549242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:18.427370093 +0000 UTC m=+1845.725847847" watchObservedRunningTime="2026-07-29 21:34:18.428549242 +0000 UTC m=+1845.727027001"
Jul 29 21:34:24.391351 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:24.391317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:29.387469 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:29.387441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:29.387948 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:29.387515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:46.949025 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:46.948985    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:46.949525 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:46.949247    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" podUID="45907e52-9928-47b3-820c-79b4fcf96638" containerName="maas-api" containerID="cri-o://7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301" gracePeriod=30
Jul 29 21:34:47.002276 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.002241    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:47.002557 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.002519    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" podUID="6019a732-1109-4dfa-bf39-8a9be727e226" containerName="payload-processing" containerID="cri-o://038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e" gracePeriod=30
Jul 29 21:34:47.009447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.009404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:47.009675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.009631    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" podUID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" containerName="payload-pre-processing" containerID="cri-o://a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53" gracePeriod=30
Jul 29 21:34:47.295052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.295025    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:47.298855 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.298829    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:47.321251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.321200    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:47.445663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtqvq\" (UniqueName: \"kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq\") pod \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t7fc\" (UniqueName: \"kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc\") pod \"6019a732-1109-4dfa-bf39-8a9be727e226\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445725    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca\") pod \"45907e52-9928-47b3-820c-79b4fcf96638\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l24jt\" (UniqueName: \"kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt\") pod \"45907e52-9928-47b3-820c-79b4fcf96638\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume\") pod \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\" (UID: \"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445835    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") pod \"45907e52-9928-47b3-820c-79b4fcf96638\" (UID: \"45907e52-9928-47b3-820c-79b4fcf96638\") "
Jul 29 21:34:47.445873 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.445873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume\") pod \"6019a732-1109-4dfa-bf39-8a9be727e226\" (UID: \"6019a732-1109-4dfa-bf39-8a9be727e226\") "
Jul 29 21:34:47.446210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.446193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45907e52-9928-47b3-820c-79b4fcf96638" (UID: "45907e52-9928-47b3-820c-79b4fcf96638"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:34:47.446301 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.446270    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45907e52-9928-47b3-820c-79b4fcf96638-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.446535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.446511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6019a732-1109-4dfa-bf39-8a9be727e226" (UID: "6019a732-1109-4dfa-bf39-8a9be727e226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:34:47.446650 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.446551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" (UID: "1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:34:47.448273 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.448249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc" (OuterVolumeSpecName: "kube-api-access-5t7fc") pod "6019a732-1109-4dfa-bf39-8a9be727e226" (UID: "6019a732-1109-4dfa-bf39-8a9be727e226"). InnerVolumeSpecName "kube-api-access-5t7fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:34:47.448410 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.448395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq" (OuterVolumeSpecName: "kube-api-access-qtqvq") pod "1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" (UID: "1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f"). InnerVolumeSpecName "kube-api-access-qtqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:34:47.448836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.448819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt" (OuterVolumeSpecName: "kube-api-access-l24jt") pod "45907e52-9928-47b3-820c-79b4fcf96638" (UID: "45907e52-9928-47b3-820c-79b4fcf96638"). InnerVolumeSpecName "kube-api-access-l24jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:34:47.448904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.448845    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45907e52-9928-47b3-820c-79b4fcf96638" (UID: "45907e52-9928-47b3-820c-79b4fcf96638"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:34:47.547906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547819    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45907e52-9928-47b3-820c-79b4fcf96638-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.547906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547855    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6019a732-1109-4dfa-bf39-8a9be727e226-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.547906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547870    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtqvq\" (UniqueName: \"kubernetes.io/projected/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-kube-api-access-qtqvq\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.547906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7fc\" (UniqueName: \"kubernetes.io/projected/6019a732-1109-4dfa-bf39-8a9be727e226-kube-api-access-5t7fc\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.547906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547900    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24jt\" (UniqueName: \"kubernetes.io/projected/45907e52-9928-47b3-820c-79b4fcf96638-kube-api-access-l24jt\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.548182 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.547925    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:34:47.553458 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.553422    2581 generic.go:356] "Generic (PLEG): container finished" podID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" containerID="a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53" exitCode=0
Jul 29 21:34:47.553602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.553472    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"
Jul 29 21:34:47.553602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.553506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" event={"ID":"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f","Type":"ContainerDied","Data":"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"}
Jul 29 21:34:47.553602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.553567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd" event={"ID":"1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f","Type":"ContainerDied","Data":"e167dd753d021ee2ff3bcc4992d63f2c18017aed8ef910d938d104588224f804"}
Jul 29 21:34:47.553602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.553595    2581 scope.go:117] "RemoveContainer" containerID="a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"
Jul 29 21:34:47.555244 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.555206    2581 generic.go:356] "Generic (PLEG): container finished" podID="6019a732-1109-4dfa-bf39-8a9be727e226" containerID="038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e" exitCode=0
Jul 29 21:34:47.555348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.555268    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"
Jul 29 21:34:47.555451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.555397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" event={"ID":"6019a732-1109-4dfa-bf39-8a9be727e226","Type":"ContainerDied","Data":"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"}
Jul 29 21:34:47.555579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.555486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl" event={"ID":"6019a732-1109-4dfa-bf39-8a9be727e226","Type":"ContainerDied","Data":"2c50686c4a13da5fa3c46b68fcc3391a4ba550dbb62b06997c9eff4eb9371ba2"}
Jul 29 21:34:47.557274 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.557248    2581 generic.go:356] "Generic (PLEG): container finished" podID="45907e52-9928-47b3-820c-79b4fcf96638" containerID="7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301" exitCode=0
Jul 29 21:34:47.557400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.557296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" event={"ID":"45907e52-9928-47b3-820c-79b4fcf96638","Type":"ContainerDied","Data":"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"}
Jul 29 21:34:47.557400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.557303    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"
Jul 29 21:34:47.557400 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.557338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4" event={"ID":"45907e52-9928-47b3-820c-79b4fcf96638","Type":"ContainerDied","Data":"f9d4d6843e4499fa9129df92f1069e7b58251bff938505249c8664b9e4d4d1f0"}
Jul 29 21:34:47.566268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.565982    2581 scope.go:117] "RemoveContainer" containerID="a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"
Jul 29 21:34:47.567103 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:47.566645    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53\": container with ID starting with a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53 not found: ID does not exist" containerID="a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"
Jul 29 21:34:47.567103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.566875    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53"} err="failed to get container status \"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53\": rpc error: code = NotFound desc = could not find container \"a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53\": container with ID starting with a9f163bbd735784241eb1c9143dc65a8239f79cdd74ed0f17a30b7b03187dd53 not found: ID does not exist"
Jul 29 21:34:47.567103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.566916    2581 scope.go:117] "RemoveContainer" containerID="038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"
Jul 29 21:34:47.582609 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.582587    2581 scope.go:117] "RemoveContainer" containerID="038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"
Jul 29 21:34:47.582954 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:47.582934    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e\": container with ID starting with 038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e not found: ID does not exist" containerID="038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"
Jul 29 21:34:47.583054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.582962    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e"} err="failed to get container status \"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e\": rpc error: code = NotFound desc = could not find container \"038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e\": container with ID starting with 038164d2c062c0d74a6d4bd9b228d7d2afc797a2bd9da1fdac75e918fd71ed1e not found: ID does not exist"
Jul 29 21:34:47.583054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.582980    2581 scope.go:117] "RemoveContainer" containerID="7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"
Jul 29 21:34:47.591665 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.591643    2581 scope.go:117] "RemoveContainer" containerID="7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"
Jul 29 21:34:47.592082 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:47.592060    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301\": container with ID starting with 7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301 not found: ID does not exist" containerID="7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"
Jul 29 21:34:47.592188 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.592089    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301"} err="failed to get container status \"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301\": rpc error: code = NotFound desc = could not find container \"7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301\": container with ID starting with 7e67ecff1088dcbe9ef0340fc83944345c8325b0520c9fa643303bdb4d595301 not found: ID does not exist"
Jul 29 21:34:47.594271 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.594250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:47.596629 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.596609    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80781434-5c57db69b8-t44bd"]
Jul 29 21:34:47.616354 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.616327    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:47.618414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.618395    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80781434-68996f7f88-mtwj4"]
Jul 29 21:34:47.631661 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.631628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:47.633620 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:47.633602    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80781434-c9b8db8db-lg5rl"]
Jul 29 21:34:49.188142 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:49.188111    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" path="/var/lib/kubelet/pods/1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f/volumes"
Jul 29 21:34:49.188531 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:49.188480    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45907e52-9928-47b3-820c-79b4fcf96638" path="/var/lib/kubelet/pods/45907e52-9928-47b3-820c-79b4fcf96638/volumes"
Jul 29 21:34:49.188844 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:49.188829    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6019a732-1109-4dfa-bf39-8a9be727e226" path="/var/lib/kubelet/pods/6019a732-1109-4dfa-bf39-8a9be727e226/volumes"
Jul 29 21:34:51.624066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624035    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:34:51.624501 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45907e52-9928-47b3-820c-79b4fcf96638" containerName="maas-api"
Jul 29 21:34:51.624501 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624491    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45907e52-9928-47b3-820c-79b4fcf96638" containerName="maas-api"
Jul 29 21:34:51.624624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624511    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6019a732-1109-4dfa-bf39-8a9be727e226" containerName="payload-processing"
Jul 29 21:34:51.624624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624524    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6019a732-1109-4dfa-bf39-8a9be727e226" containerName="payload-processing"
Jul 29 21:34:51.624624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624548    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" containerName="payload-pre-processing"
Jul 29 21:34:51.624624 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624558    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" containerName="payload-pre-processing"
Jul 29 21:34:51.624819 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624660    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6019a732-1109-4dfa-bf39-8a9be727e226" containerName="payload-processing"
Jul 29 21:34:51.624819 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624672    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45907e52-9928-47b3-820c-79b4fcf96638" containerName="maas-api"
Jul 29 21:34:51.624819 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.624683    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1260a5fa-66a4-4a9c-9714-1f1ec4d7a06f" containerName="payload-pre-processing"
Jul 29 21:34:51.629396 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.629377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.632286 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.632267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f6f4f61f-serving-cert\""
Jul 29 21:34:51.645859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.645831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:34:51.683605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.683574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv886\" (UniqueName: \"kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.683605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.683611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.683904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.683795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.785011 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.784974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.785265 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.785030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv886\" (UniqueName: \"kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.785265 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.785054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.785265 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:51.785113    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-serving-cert: secret "maas-api-e2e-migrate-f6f4f61f-serving-cert" not found
Jul 29 21:34:51.785265 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:34:51.785180    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls podName:a60470a8-9452-4ee8-a6ab-3b265016444a nodeName:}" failed. No retries permitted until 2026-07-29 21:34:52.285162434 +0000 UTC m=+1879.583640167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls") pod "maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" (UID: "a60470a8-9452-4ee8-a6ab-3b265016444a") : secret "maas-api-e2e-migrate-f6f4f61f-serving-cert" not found
Jul 29 21:34:51.785647 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.785630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:51.794562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:51.794532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv886\" (UniqueName: \"kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:52.255828 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.255782    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:34:52.259622 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.259594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.262158 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.262134    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f6f4f61f\""
Jul 29 21:34:52.262325 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.262159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f6f4f61f-dockercfg-8mnjp\""
Jul 29 21:34:52.271715 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.271665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:34:52.285446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.285398    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:34:52.293545 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.292158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.293545 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.292272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:52.293545 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.292397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7p2\" (UniqueName: \"kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.293545 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.292554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.296576 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.296553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") pod \"maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:52.301915 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.301884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:34:52.393375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.393339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7p2\" (UniqueName: \"kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.393535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.393398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.393535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.393439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsgq\" (UniqueName: \"kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.393535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.393473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.394247 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.394211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.403215 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.403171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7p2\" (UniqueName: \"kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2\") pod \"payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.494414 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.494380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.494597 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.494489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsgq\" (UniqueName: \"kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.495159 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.495136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.504503 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.504453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsgq\" (UniqueName: \"kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq\") pod \"payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") " pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.540394 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.540304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:52.571815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.571776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:52.621751 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.621624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:52.751315 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.751264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:34:52.793744 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.793662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:34:52.795126 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:34:52.795098    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c403f1b_cf48_4efc_b8df_f40a87c48e74.slice/crio-46e5f6b28e8421cd2069d785affdab385e620a5b9f60034877b0727d66d8c28c WatchSource:0}: Error finding container 46e5f6b28e8421cd2069d785affdab385e620a5b9f60034877b0727d66d8c28c: Status 404 returned error can't find the container with id 46e5f6b28e8421cd2069d785affdab385e620a5b9f60034877b0727d66d8c28c
Jul 29 21:34:52.826563 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:52.826533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:34:52.829059 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:34:52.829034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode798bfe3_b7ef_414c_b0db_9d4883aae448.slice/crio-c3d61cb4e733e0bee68180c7aa40b72bf241ae6867d79a6f9b6ccb21bac5dc5b WatchSource:0}: Error finding container c3d61cb4e733e0bee68180c7aa40b72bf241ae6867d79a6f9b6ccb21bac5dc5b: Status 404 returned error can't find the container with id c3d61cb4e733e0bee68180c7aa40b72bf241ae6867d79a6f9b6ccb21bac5dc5b
Jul 29 21:34:53.598361 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.598325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" event={"ID":"7c403f1b-cf48-4efc-b8df-f40a87c48e74","Type":"ContainerStarted","Data":"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"}
Jul 29 21:34:53.598547 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.598384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" event={"ID":"7c403f1b-cf48-4efc-b8df-f40a87c48e74","Type":"ContainerStarted","Data":"46e5f6b28e8421cd2069d785affdab385e620a5b9f60034877b0727d66d8c28c"}
Jul 29 21:34:53.598619 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.598564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:34:53.600105 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.600079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" event={"ID":"a60470a8-9452-4ee8-a6ab-3b265016444a","Type":"ContainerStarted","Data":"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"}
Jul 29 21:34:53.600211 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.600127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" event={"ID":"a60470a8-9452-4ee8-a6ab-3b265016444a","Type":"ContainerStarted","Data":"06c52f663439d8b489faeb7d974d85300e4396ff0575d5c9a311902960dddbd2"}
Jul 29 21:34:53.600211 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.600158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:34:53.601540 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.601522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" event={"ID":"e798bfe3-b7ef-414c-b0db-9d4883aae448","Type":"ContainerStarted","Data":"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"}
Jul 29 21:34:53.601626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.601553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" event={"ID":"e798bfe3-b7ef-414c-b0db-9d4883aae448","Type":"ContainerStarted","Data":"c3d61cb4e733e0bee68180c7aa40b72bf241ae6867d79a6f9b6ccb21bac5dc5b"}
Jul 29 21:34:53.601674 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.601650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:34:53.624968 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.624913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" podStartSLOduration=1.624895947 podStartE2EDuration="1.624895947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:53.624157482 +0000 UTC m=+1880.922635238" watchObservedRunningTime="2026-07-29 21:34:53.624895947 +0000 UTC m=+1880.923373701"
Jul 29 21:34:53.646877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.646813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" podStartSLOduration=1.646796314 podStartE2EDuration="1.646796314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:53.645289102 +0000 UTC m=+1880.943766858" watchObservedRunningTime="2026-07-29 21:34:53.646796314 +0000 UTC m=+1880.945274069"
Jul 29 21:34:53.662582 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:53.662519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" podStartSLOduration=1.9127108229999998 podStartE2EDuration="2.662497106s" totalImagesPullingTime="749.786283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:34:53.661768801 +0000 UTC m=+1880.960246556" watchObservedRunningTime="2026-07-29 21:34:53.662497106 +0000 UTC m=+1880.960974862"
Jul 29 21:34:59.614155 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:34:59.614124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:35:04.610084 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:04.610051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:35:04.610499 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:04.610145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:35:22.950634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:22.950537    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:35:22.951171 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:22.950773    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" podUID="a60470a8-9452-4ee8-a6ab-3b265016444a" containerName="maas-api" containerID="cri-o://c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52" gracePeriod=30
Jul 29 21:35:23.005474 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.005434    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:35:23.005690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.005648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" podUID="e798bfe3-b7ef-414c-b0db-9d4883aae448" containerName="payload-processing" containerID="cri-o://cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530" gracePeriod=30
Jul 29 21:35:23.012221 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.012176    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:35:23.012731 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.012646    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" podUID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" containerName="payload-pre-processing" containerID="cri-o://8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd" gracePeriod=30
Jul 29 21:35:23.295365 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.295338    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:35:23.299546 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.299519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:35:23.315290 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.315260    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:35:23.384287 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv886\" (UniqueName: \"kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886\") pod \"a60470a8-9452-4ee8-a6ab-3b265016444a\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") "
Jul 29 21:35:23.384461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") pod \"a60470a8-9452-4ee8-a6ab-3b265016444a\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") "
Jul 29 21:35:23.384517 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume\") pod \"e798bfe3-b7ef-414c-b0db-9d4883aae448\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") "
Jul 29 21:35:23.384583 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca\") pod \"a60470a8-9452-4ee8-a6ab-3b265016444a\" (UID: \"a60470a8-9452-4ee8-a6ab-3b265016444a\") "
Jul 29 21:35:23.384633 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhsgq\" (UniqueName: \"kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq\") pod \"e798bfe3-b7ef-414c-b0db-9d4883aae448\" (UID: \"e798bfe3-b7ef-414c-b0db-9d4883aae448\") "
Jul 29 21:35:23.384915 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e798bfe3-b7ef-414c-b0db-9d4883aae448" (UID: "e798bfe3-b7ef-414c-b0db-9d4883aae448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:23.385047 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.384937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a60470a8-9452-4ee8-a6ab-3b265016444a" (UID: "a60470a8-9452-4ee8-a6ab-3b265016444a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:23.386844 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.386818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq" (OuterVolumeSpecName: "kube-api-access-nhsgq") pod "e798bfe3-b7ef-414c-b0db-9d4883aae448" (UID: "e798bfe3-b7ef-414c-b0db-9d4883aae448"). InnerVolumeSpecName "kube-api-access-nhsgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:23.387248 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.387221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a60470a8-9452-4ee8-a6ab-3b265016444a" (UID: "a60470a8-9452-4ee8-a6ab-3b265016444a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:35:23.387248 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.387220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886" (OuterVolumeSpecName: "kube-api-access-zv886") pod "a60470a8-9452-4ee8-a6ab-3b265016444a" (UID: "a60470a8-9452-4ee8-a6ab-3b265016444a"). InnerVolumeSpecName "kube-api-access-zv886". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:23.485884 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.485780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v7p2\" (UniqueName: \"kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2\") pod \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") "
Jul 29 21:35:23.485884 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.485859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume\") pod \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\" (UID: \"7c403f1b-cf48-4efc-b8df-f40a87c48e74\") "
Jul 29 21:35:23.486124 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486106    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60470a8-9452-4ee8-a6ab-3b265016444a-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.486190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486131    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e798bfe3-b7ef-414c-b0db-9d4883aae448-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.486190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486146    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60470a8-9452-4ee8-a6ab-3b265016444a-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.486190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486160    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsgq\" (UniqueName: \"kubernetes.io/projected/e798bfe3-b7ef-414c-b0db-9d4883aae448-kube-api-access-nhsgq\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.486190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486177    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv886\" (UniqueName: \"kubernetes.io/projected/a60470a8-9452-4ee8-a6ab-3b265016444a-kube-api-access-zv886\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.486333 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.486285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c403f1b-cf48-4efc-b8df-f40a87c48e74" (UID: "7c403f1b-cf48-4efc-b8df-f40a87c48e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:23.488282 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.488259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2" (OuterVolumeSpecName: "kube-api-access-5v7p2") pod "7c403f1b-cf48-4efc-b8df-f40a87c48e74" (UID: "7c403f1b-cf48-4efc-b8df-f40a87c48e74"). InnerVolumeSpecName "kube-api-access-5v7p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:23.587265 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.587221    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v7p2\" (UniqueName: \"kubernetes.io/projected/7c403f1b-cf48-4efc-b8df-f40a87c48e74-kube-api-access-5v7p2\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.587265 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.587258    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c403f1b-cf48-4efc-b8df-f40a87c48e74-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:23.773801 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.773686    2581 generic.go:356] "Generic (PLEG): container finished" podID="e798bfe3-b7ef-414c-b0db-9d4883aae448" containerID="cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530" exitCode=0
Jul 29 21:35:23.773801 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.773749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"
Jul 29 21:35:23.773801 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.773771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" event={"ID":"e798bfe3-b7ef-414c-b0db-9d4883aae448","Type":"ContainerDied","Data":"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"}
Jul 29 21:35:23.774089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.773828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g" event={"ID":"e798bfe3-b7ef-414c-b0db-9d4883aae448","Type":"ContainerDied","Data":"c3d61cb4e733e0bee68180c7aa40b72bf241ae6867d79a6f9b6ccb21bac5dc5b"}
Jul 29 21:35:23.774089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.773857    2581 scope.go:117] "RemoveContainer" containerID="cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"
Jul 29 21:35:23.775387 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.775357    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" containerID="8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd" exitCode=0
Jul 29 21:35:23.775530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.775402    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"
Jul 29 21:35:23.775530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.775455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" event={"ID":"7c403f1b-cf48-4efc-b8df-f40a87c48e74","Type":"ContainerDied","Data":"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"}
Jul 29 21:35:23.775530 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.775511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8" event={"ID":"7c403f1b-cf48-4efc-b8df-f40a87c48e74","Type":"ContainerDied","Data":"46e5f6b28e8421cd2069d785affdab385e620a5b9f60034877b0727d66d8c28c"}
Jul 29 21:35:23.777091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.777063    2581 generic.go:356] "Generic (PLEG): container finished" podID="a60470a8-9452-4ee8-a6ab-3b265016444a" containerID="c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52" exitCode=0
Jul 29 21:35:23.777209 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.777097    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"
Jul 29 21:35:23.777209 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.777150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" event={"ID":"a60470a8-9452-4ee8-a6ab-3b265016444a","Type":"ContainerDied","Data":"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"}
Jul 29 21:35:23.777209 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.777204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7" event={"ID":"a60470a8-9452-4ee8-a6ab-3b265016444a","Type":"ContainerDied","Data":"06c52f663439d8b489faeb7d974d85300e4396ff0575d5c9a311902960dddbd2"}
Jul 29 21:35:23.784674 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.784648    2581 scope.go:117] "RemoveContainer" containerID="cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"
Jul 29 21:35:23.785052 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:23.785030    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530\": container with ID starting with cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530 not found: ID does not exist" containerID="cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"
Jul 29 21:35:23.785145 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.785062    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530"} err="failed to get container status \"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530\": rpc error: code = NotFound desc = could not find container \"cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530\": container with ID starting with cd5eae53cd00210358b4b2662c09e51601b14f56c6445b42f148ed7bd0b05530 not found: ID does not exist"
Jul 29 21:35:23.785145 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.785081    2581 scope.go:117] "RemoveContainer" containerID="8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"
Jul 29 21:35:23.794267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.794247    2581 scope.go:117] "RemoveContainer" containerID="8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"
Jul 29 21:35:23.794566 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:23.794546    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd\": container with ID starting with 8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd not found: ID does not exist" containerID="8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"
Jul 29 21:35:23.794636 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.794575    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd"} err="failed to get container status \"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd\": rpc error: code = NotFound desc = could not find container \"8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd\": container with ID starting with 8fb7f448171ba3a9aa76dfec3121a3de5e1a2c66cdf3ac4cf6d74d00539a71fd not found: ID does not exist"
Jul 29 21:35:23.794636 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.794594    2581 scope.go:117] "RemoveContainer" containerID="c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"
Jul 29 21:35:23.803401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.803375    2581 scope.go:117] "RemoveContainer" containerID="c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"
Jul 29 21:35:23.803749 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:23.803728    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52\": container with ID starting with c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52 not found: ID does not exist" containerID="c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"
Jul 29 21:35:23.803821 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.803759    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52"} err="failed to get container status \"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52\": rpc error: code = NotFound desc = could not find container \"c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52\": container with ID starting with c1a79f4d38a20b823e54ebc0c4a1e00176a24c107109041cfee542d06c8eed52 not found: ID does not exist"
Jul 29 21:35:23.807325 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.807298    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:35:23.812541 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.812511    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f6f4f61f-66756785c7-gnb4g"]
Jul 29 21:35:23.823155 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.823120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:35:23.827066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.827034    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f6f4f61f-7c89b65d69-rkdl7"]
Jul 29 21:35:23.839339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.839306    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:35:23.841541 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:23.841520    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f6f4f61f-6856b59f84-lw8v8"]
Jul 29 21:35:25.193560 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:25.193514    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" path="/var/lib/kubelet/pods/7c403f1b-cf48-4efc-b8df-f40a87c48e74/volumes"
Jul 29 21:35:25.194127 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:25.194105    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60470a8-9452-4ee8-a6ab-3b265016444a" path="/var/lib/kubelet/pods/a60470a8-9452-4ee8-a6ab-3b265016444a/volumes"
Jul 29 21:35:25.194526 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:25.194506    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e798bfe3-b7ef-414c-b0db-9d4883aae448" path="/var/lib/kubelet/pods/e798bfe3-b7ef-414c-b0db-9d4883aae448/volumes"
Jul 29 21:35:27.637435 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.637386    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:27.637989 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.637971    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" containerName="payload-pre-processing"
Jul 29 21:35:27.638044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.637994    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" containerName="payload-pre-processing"
Jul 29 21:35:27.638044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638026    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60470a8-9452-4ee8-a6ab-3b265016444a" containerName="maas-api"
Jul 29 21:35:27.638044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638035    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60470a8-9452-4ee8-a6ab-3b265016444a" containerName="maas-api"
Jul 29 21:35:27.638153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e798bfe3-b7ef-414c-b0db-9d4883aae448" containerName="payload-processing"
Jul 29 21:35:27.638153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638057    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e798bfe3-b7ef-414c-b0db-9d4883aae448" containerName="payload-processing"
Jul 29 21:35:27.638153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638143    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e798bfe3-b7ef-414c-b0db-9d4883aae448" containerName="payload-processing"
Jul 29 21:35:27.638262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638155    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c403f1b-cf48-4efc-b8df-f40a87c48e74" containerName="payload-pre-processing"
Jul 29 21:35:27.638262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.638166    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60470a8-9452-4ee8-a6ab-3b265016444a" containerName="maas-api"
Jul 29 21:35:27.641810 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.641790    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.644047 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.644023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d8fc281-serving-cert\""
Jul 29 21:35:27.651654 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.651629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:27.727196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.727150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ln7\" (UniqueName: \"kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.727384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.727203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.727384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.727355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.828280 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.828229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ln7\" (UniqueName: \"kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.828280 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.828285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.828503 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.828396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.828561 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:27.828539    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-serving-cert: secret "maas-api-e2e-ait-0d8fc281-serving-cert" not found
Jul 29 21:35:27.828616 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:27.828605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls podName:0869da00-9730-4b80-b378-87a1623540b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:35:28.328589856 +0000 UTC m=+1915.627067592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls") pod "maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" (UID: "0869da00-9730-4b80-b378-87a1623540b1") : secret "maas-api-e2e-ait-0d8fc281-serving-cert" not found
Jul 29 21:35:27.829085 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.829063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:27.839069 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:27.839040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ln7\" (UniqueName: \"kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:28.333589 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.333549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:28.333809 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:28.333726    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-serving-cert: secret "maas-api-e2e-ait-0d8fc281-serving-cert" not found
Jul 29 21:35:28.333809 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:28.333801    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls podName:0869da00-9730-4b80-b378-87a1623540b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:35:29.333782874 +0000 UTC m=+1916.632260607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls") pod "maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" (UID: "0869da00-9730-4b80-b378-87a1623540b1") : secret "maas-api-e2e-ait-0d8fc281-serving-cert" not found
Jul 29 21:35:28.401016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.400977    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:28.404740 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.404683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.407140 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.407115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0d8fc281\""
Jul 29 21:35:28.417122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.417094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:28.429476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.429440    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:28.433449 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.433419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.435914 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.435886    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0d8fc281-dockercfg-vg6wk\""
Jul 29 21:35:28.446989 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.446954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:28.535571 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.535532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjp6\" (UniqueName: \"kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.535780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.535611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.535780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.535682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.535780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.535764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5fp\" (UniqueName: \"kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.637183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.637149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.637389 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.637222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.637389 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.637254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5fp\" (UniqueName: \"kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.637389 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.637284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjp6\" (UniqueName: \"kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.638289 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.638261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.638365 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.638312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.646174 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.646146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5fp\" (UniqueName: \"kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp\") pod \"payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") " pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.646300 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.646240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjp6\" (UniqueName: \"kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6\") pod \"payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.716648 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.716614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:28.749667 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.749633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:28.899635 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.899557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:28.933056 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:28.933025    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:28.940085 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:35:28.940055    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff12c6c_d034_4005_bd57_90a5e8a5afbf.slice/crio-8c9bd1153634a0a9541afec5445e84f86c7b154c8a30c420f9e2b167649bb7a0 WatchSource:0}: Error finding container 8c9bd1153634a0a9541afec5445e84f86c7b154c8a30c420f9e2b167649bb7a0: Status 404 returned error can't find the container with id 8c9bd1153634a0a9541afec5445e84f86c7b154c8a30c420f9e2b167649bb7a0
Jul 29 21:35:29.345014 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.344978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:29.347725 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.347671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:29.455921 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.455880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:29.615000 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.614968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:29.616774 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:35:29.616750    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0869da00_9730_4b80_b378_87a1623540b1.slice/crio-c8fd7a039b15b45d8e780afd43b44de86c9abb0108599da3d6951a9257f1ab50 WatchSource:0}: Error finding container c8fd7a039b15b45d8e780afd43b44de86c9abb0108599da3d6951a9257f1ab50: Status 404 returned error can't find the container with id c8fd7a039b15b45d8e780afd43b44de86c9abb0108599da3d6951a9257f1ab50
Jul 29 21:35:29.816064 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.816016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" event={"ID":"eff12c6c-d034-4005-bd57-90a5e8a5afbf","Type":"ContainerStarted","Data":"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"}
Jul 29 21:35:29.816566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.816085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" event={"ID":"eff12c6c-d034-4005-bd57-90a5e8a5afbf","Type":"ContainerStarted","Data":"8c9bd1153634a0a9541afec5445e84f86c7b154c8a30c420f9e2b167649bb7a0"}
Jul 29 21:35:29.816566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.816242    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:29.817523 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.817491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" event={"ID":"0869da00-9730-4b80-b378-87a1623540b1","Type":"ContainerStarted","Data":"c8fd7a039b15b45d8e780afd43b44de86c9abb0108599da3d6951a9257f1ab50"}
Jul 29 21:35:29.819346 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.819319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" event={"ID":"b3e30117-b037-4472-8fc3-769faf7e22f1","Type":"ContainerStarted","Data":"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"}
Jul 29 21:35:29.819490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.819370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" event={"ID":"b3e30117-b037-4472-8fc3-769faf7e22f1","Type":"ContainerStarted","Data":"ad5f14b2635becc785102b24c416323cd63a9dba58f89e6967ca60d9ebc5f399"}
Jul 29 21:35:29.819490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.819390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:29.834950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.834876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" podStartSLOduration=1.834859882 podStartE2EDuration="1.834859882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:35:29.832472157 +0000 UTC m=+1917.130949923" watchObservedRunningTime="2026-07-29 21:35:29.834859882 +0000 UTC m=+1917.133337636"
Jul 29 21:35:29.850415 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:29.850354    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" podStartSLOduration=1.850338928 podStartE2EDuration="1.850338928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:35:29.849792688 +0000 UTC m=+1917.148270436" watchObservedRunningTime="2026-07-29 21:35:29.850338928 +0000 UTC m=+1917.148816683"
Jul 29 21:35:30.826767 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:30.826731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" event={"ID":"0869da00-9730-4b80-b378-87a1623540b1","Type":"ContainerStarted","Data":"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"}
Jul 29 21:35:30.827235 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:30.827090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:30.845428 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:30.845379    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" podStartSLOduration=3.3880772390000002 podStartE2EDuration="3.845347048s" totalImagesPullingTime="457.269809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:35:30.844499248 +0000 UTC m=+1918.142977004" watchObservedRunningTime="2026-07-29 21:35:30.845347048 +0000 UTC m=+1918.143824802"
Jul 29 21:35:37.847752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:37.847717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:40.828893 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:40.828857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:40.829512 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:40.829485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:58.214397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.214359    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:58.217147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.214622    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" podUID="0869da00-9730-4b80-b378-87a1623540b1" containerName="maas-api" containerID="cri-o://6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b" gracePeriod=30
Jul 29 21:35:58.285957 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.285916    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:58.286252 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.286194    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" podUID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" containerName="payload-processing" containerID="cri-o://3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b" gracePeriod=30
Jul 29 21:35:58.294567 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.294527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:58.294837 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.294808    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" podUID="b3e30117-b037-4472-8fc3-769faf7e22f1" containerName="payload-pre-processing" containerID="cri-o://e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c" gracePeriod=30
Jul 29 21:35:58.577182 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.577153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:58.581981 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.581953    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:58.613587 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.613534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:58.689566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld5fp\" (UniqueName: \"kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp\") pod \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsjp6\" (UniqueName: \"kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6\") pod \"b3e30117-b037-4472-8fc3-769faf7e22f1\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca\") pod \"0869da00-9730-4b80-b378-87a1623540b1\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689625    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") pod \"0869da00-9730-4b80-b378-87a1623540b1\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8ln7\" (UniqueName: \"kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7\") pod \"0869da00-9730-4b80-b378-87a1623540b1\" (UID: \"0869da00-9730-4b80-b378-87a1623540b1\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume\") pod \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\" (UID: \"eff12c6c-d034-4005-bd57-90a5e8a5afbf\") "
Jul 29 21:35:58.689817 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.689775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume\") pod \"b3e30117-b037-4472-8fc3-769faf7e22f1\" (UID: \"b3e30117-b037-4472-8fc3-769faf7e22f1\") "
Jul 29 21:35:58.690220 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0869da00-9730-4b80-b378-87a1623540b1" (UID: "0869da00-9730-4b80-b378-87a1623540b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:58.690220 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eff12c6c-d034-4005-bd57-90a5e8a5afbf" (UID: "eff12c6c-d034-4005-bd57-90a5e8a5afbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:58.690355 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3e30117-b037-4472-8fc3-769faf7e22f1" (UID: "b3e30117-b037-4472-8fc3-769faf7e22f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:35:58.690528 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690505    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0869da00-9730-4b80-b378-87a1623540b1-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.690528 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690529    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff12c6c-d034-4005-bd57-90a5e8a5afbf-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.690684 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.690545    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e30117-b037-4472-8fc3-769faf7e22f1-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.692252 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.692228    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0869da00-9730-4b80-b378-87a1623540b1" (UID: "0869da00-9730-4b80-b378-87a1623540b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:35:58.692424 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.692409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6" (OuterVolumeSpecName: "kube-api-access-gsjp6") pod "b3e30117-b037-4472-8fc3-769faf7e22f1" (UID: "b3e30117-b037-4472-8fc3-769faf7e22f1"). InnerVolumeSpecName "kube-api-access-gsjp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:58.692580 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.692554    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7" (OuterVolumeSpecName: "kube-api-access-t8ln7") pod "0869da00-9730-4b80-b378-87a1623540b1" (UID: "0869da00-9730-4b80-b378-87a1623540b1"). InnerVolumeSpecName "kube-api-access-t8ln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:58.692672 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.692588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp" (OuterVolumeSpecName: "kube-api-access-ld5fp") pod "eff12c6c-d034-4005-bd57-90a5e8a5afbf" (UID: "eff12c6c-d034-4005-bd57-90a5e8a5afbf"). InnerVolumeSpecName "kube-api-access-ld5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:35:58.791590 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.791495    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5fp\" (UniqueName: \"kubernetes.io/projected/eff12c6c-d034-4005-bd57-90a5e8a5afbf-kube-api-access-ld5fp\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.791590 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.791529    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjp6\" (UniqueName: \"kubernetes.io/projected/b3e30117-b037-4472-8fc3-769faf7e22f1-kube-api-access-gsjp6\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.791590 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.791542    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0869da00-9730-4b80-b378-87a1623540b1-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:58.791590 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:58.791551    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ln7\" (UniqueName: \"kubernetes.io/projected/0869da00-9730-4b80-b378-87a1623540b1-kube-api-access-t8ln7\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:35:59.005141 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.005102    2581 generic.go:356] "Generic (PLEG): container finished" podID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" containerID="3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b" exitCode=0
Jul 29 21:35:59.005342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.005153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"
Jul 29 21:35:59.005342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.005185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" event={"ID":"eff12c6c-d034-4005-bd57-90a5e8a5afbf","Type":"ContainerDied","Data":"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"}
Jul 29 21:35:59.005342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.005268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27" event={"ID":"eff12c6c-d034-4005-bd57-90a5e8a5afbf","Type":"ContainerDied","Data":"8c9bd1153634a0a9541afec5445e84f86c7b154c8a30c420f9e2b167649bb7a0"}
Jul 29 21:35:59.005342 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.005294    2581 scope.go:117] "RemoveContainer" containerID="3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"
Jul 29 21:35:59.006595 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.006562    2581 generic.go:356] "Generic (PLEG): container finished" podID="0869da00-9730-4b80-b378-87a1623540b1" containerID="6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b" exitCode=0
Jul 29 21:35:59.006681 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.006640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"
Jul 29 21:35:59.006681 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.006650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" event={"ID":"0869da00-9730-4b80-b378-87a1623540b1","Type":"ContainerDied","Data":"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"}
Jul 29 21:35:59.006824 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.006721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx" event={"ID":"0869da00-9730-4b80-b378-87a1623540b1","Type":"ContainerDied","Data":"c8fd7a039b15b45d8e780afd43b44de86c9abb0108599da3d6951a9257f1ab50"}
Jul 29 21:35:59.008285 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.008247    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3e30117-b037-4472-8fc3-769faf7e22f1" containerID="e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c" exitCode=0
Jul 29 21:35:59.008419 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.008292    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"
Jul 29 21:35:59.008419 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.008318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" event={"ID":"b3e30117-b037-4472-8fc3-769faf7e22f1","Type":"ContainerDied","Data":"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"}
Jul 29 21:35:59.008419 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.008356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p" event={"ID":"b3e30117-b037-4472-8fc3-769faf7e22f1","Type":"ContainerDied","Data":"ad5f14b2635becc785102b24c416323cd63a9dba58f89e6967ca60d9ebc5f399"}
Jul 29 21:35:59.016352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.016333    2581 scope.go:117] "RemoveContainer" containerID="3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"
Jul 29 21:35:59.016904 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:59.016735    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b\": container with ID starting with 3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b not found: ID does not exist" containerID="3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"
Jul 29 21:35:59.016904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.016777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b"} err="failed to get container status \"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b\": rpc error: code = NotFound desc = could not find container \"3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b\": container with ID starting with 3288d8fbfdc85c9fd29833f7845180a04f97bc8c22eb28f86f27bb68ab99351b not found: ID does not exist"
Jul 29 21:35:59.016904 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.016804    2581 scope.go:117] "RemoveContainer" containerID="6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"
Jul 29 21:35:59.026157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.026139    2581 scope.go:117] "RemoveContainer" containerID="6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"
Jul 29 21:35:59.026435 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:59.026415    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b\": container with ID starting with 6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b not found: ID does not exist" containerID="6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"
Jul 29 21:35:59.026497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.026441    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b"} err="failed to get container status \"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b\": rpc error: code = NotFound desc = could not find container \"6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b\": container with ID starting with 6b40322b456d72b446a186e26f1e52925faf78fdec429a702b61a6674bdf765b not found: ID does not exist"
Jul 29 21:35:59.026497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.026458    2581 scope.go:117] "RemoveContainer" containerID="e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"
Jul 29 21:35:59.033776 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.033747    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:59.035214 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.035193    2581 scope.go:117] "RemoveContainer" containerID="e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"
Jul 29 21:35:59.035501 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:35:59.035477    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c\": container with ID starting with e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c not found: ID does not exist" containerID="e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"
Jul 29 21:35:59.035564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.035514    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c"} err="failed to get container status \"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c\": rpc error: code = NotFound desc = could not find container \"e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c\": container with ID starting with e8c9bb76ace1ac78801eaf154cf4fdbe29878db63dd5b743393ac97b47ef496c not found: ID does not exist"
Jul 29 21:35:59.039746 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.039723    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0d8fc281-589b85d864-w9g27"]
Jul 29 21:35:59.049930 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.049863    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:59.056303 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.056275    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0d8fc281-5c65c6bfd4-spk2p"]
Jul 29 21:35:59.068840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.068691    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:59.072449 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.072415    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d8fc281-7cb8f58785-jb8wx"]
Jul 29 21:35:59.188864 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.188825    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0869da00-9730-4b80-b378-87a1623540b1" path="/var/lib/kubelet/pods/0869da00-9730-4b80-b378-87a1623540b1/volumes"
Jul 29 21:35:59.189309 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.189274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e30117-b037-4472-8fc3-769faf7e22f1" path="/var/lib/kubelet/pods/b3e30117-b037-4472-8fc3-769faf7e22f1/volumes"
Jul 29 21:35:59.189771 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:35:59.189734    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" path="/var/lib/kubelet/pods/eff12c6c-d034-4005-bd57-90a5e8a5afbf/volumes"
Jul 29 21:36:03.911109 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911071    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e30117-b037-4472-8fc3-769faf7e22f1" containerName="payload-pre-processing"
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911455    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e30117-b037-4472-8fc3-769faf7e22f1" containerName="payload-pre-processing"
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911463    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0869da00-9730-4b80-b378-87a1623540b1" containerName="maas-api"
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911469    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0869da00-9730-4b80-b378-87a1623540b1" containerName="maas-api"
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911487    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" containerName="payload-processing"
Jul 29 21:36:03.911507 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911493    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" containerName="payload-processing"
Jul 29 21:36:03.911727 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911550    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff12c6c-d034-4005-bd57-90a5e8a5afbf" containerName="payload-processing"
Jul 29 21:36:03.911727 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e30117-b037-4472-8fc3-769faf7e22f1" containerName="payload-pre-processing"
Jul 29 21:36:03.911727 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.911564    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0869da00-9730-4b80-b378-87a1623540b1" containerName="maas-api"
Jul 29 21:36:03.916167 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.916129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:03.918488 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.918463    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-592dab8c-serving-cert\""
Jul 29 21:36:03.926557 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:03.926527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:04.042461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.042427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.042461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.042462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xqv\" (UniqueName: \"kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.042709 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.042485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.143489 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.143451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.143489 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.143491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xqv\" (UniqueName: \"kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.143830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.143515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.143830 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:36:04.143636    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-serving-cert: secret "maas-api-e2e-derive-592dab8c-serving-cert" not found
Jul 29 21:36:04.143830 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:36:04.143746    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls podName:e5618028-a7f4-4cb1-97b8-1888a9bb6a0b nodeName:}" failed. No retries permitted until 2026-07-29 21:36:04.643722745 +0000 UTC m=+1951.942200493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls") pod "maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" (UID: "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b") : secret "maas-api-e2e-derive-592dab8c-serving-cert" not found
Jul 29 21:36:04.144193 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.144174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.152913 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.152890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xqv\" (UniqueName: \"kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.591268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.591229    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:04.594906 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.594885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.597469 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.597448    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-592dab8c\""
Jul 29 21:36:04.597576 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.597531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-592dab8c-dockercfg-nnv7k\""
Jul 29 21:36:04.611206 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.611167    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:04.623378 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.623340    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:04.626925 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.626902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.638625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.638598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:04.648006 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.647974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.648212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.648028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.648212 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.648152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfkc\" (UniqueName: \"kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.650646 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.650616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") pod \"maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.749196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.749159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njhtr\" (UniqueName: \"kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.749196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.749209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.749482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.749247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.749482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.749296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfkc\" (UniqueName: \"kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.750044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.750019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.757439 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.757412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfkc\" (UniqueName: \"kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc\") pod \"payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.828102 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.828062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:04.850655 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.850625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.850859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.850781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njhtr\" (UniqueName: \"kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.851604 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.851578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.860584 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.860553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njhtr\" (UniqueName: \"kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr\") pod \"payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") " pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.907117 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.907072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:04.939160 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.939120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:04.984481 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:04.984113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:04.988146 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:36:04.988102    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5618028_a7f4_4cb1_97b8_1888a9bb6a0b.slice/crio-02296bfd7cba6bf3f7c9a65ed02de4a992b41e0ff39c558c53e7c4dce8d102ca WatchSource:0}: Error finding container 02296bfd7cba6bf3f7c9a65ed02de4a992b41e0ff39c558c53e7c4dce8d102ca: Status 404 returned error can't find the container with id 02296bfd7cba6bf3f7c9a65ed02de4a992b41e0ff39c558c53e7c4dce8d102ca
Jul 29 21:36:05.049561 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:05.049521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" event={"ID":"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b","Type":"ContainerStarted","Data":"02296bfd7cba6bf3f7c9a65ed02de4a992b41e0ff39c558c53e7c4dce8d102ca"}
Jul 29 21:36:05.090627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:05.090596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:05.090931 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:36:05.090901    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96789efd_a432_45d2_ac4f_34275ccb6be7.slice/crio-fabd951999e2157c408d89e1d42b5c341942c09ad7f76c928b7dbb3b9297ce73 WatchSource:0}: Error finding container fabd951999e2157c408d89e1d42b5c341942c09ad7f76c928b7dbb3b9297ce73: Status 404 returned error can't find the container with id fabd951999e2157c408d89e1d42b5c341942c09ad7f76c928b7dbb3b9297ce73
Jul 29 21:36:05.117856 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:05.117831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:05.122968 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:36:05.122935    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d0f57a_cfb3_4377_8444_704f8354273c.slice/crio-8c3e5ceb2994d66f5b712ba998ab7af8f3e1c8b902fdd14d4c42ea6a1f2816a5 WatchSource:0}: Error finding container 8c3e5ceb2994d66f5b712ba998ab7af8f3e1c8b902fdd14d4c42ea6a1f2816a5: Status 404 returned error can't find the container with id 8c3e5ceb2994d66f5b712ba998ab7af8f3e1c8b902fdd14d4c42ea6a1f2816a5
Jul 29 21:36:06.056779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.056735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" event={"ID":"06d0f57a-cfb3-4377-8444-704f8354273c","Type":"ContainerStarted","Data":"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"}
Jul 29 21:36:06.057318 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.056808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" event={"ID":"06d0f57a-cfb3-4377-8444-704f8354273c","Type":"ContainerStarted","Data":"8c3e5ceb2994d66f5b712ba998ab7af8f3e1c8b902fdd14d4c42ea6a1f2816a5"}
Jul 29 21:36:06.057318 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.056878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:06.059145 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.059111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" event={"ID":"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b","Type":"ContainerStarted","Data":"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"}
Jul 29 21:36:06.059380 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.059361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:06.061302 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.061265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" event={"ID":"96789efd-a432-45d2-ac4f-34275ccb6be7","Type":"ContainerStarted","Data":"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"}
Jul 29 21:36:06.061431 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.061314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" event={"ID":"96789efd-a432-45d2-ac4f-34275ccb6be7","Type":"ContainerStarted","Data":"fabd951999e2157c408d89e1d42b5c341942c09ad7f76c928b7dbb3b9297ce73"}
Jul 29 21:36:06.061571 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.061545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:06.074334 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.074282    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" podStartSLOduration=2.074263948 podStartE2EDuration="2.074263948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:36:06.072002468 +0000 UTC m=+1953.370480235" watchObservedRunningTime="2026-07-29 21:36:06.074263948 +0000 UTC m=+1953.372741706"
Jul 29 21:36:06.089517 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.089453    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" podStartSLOduration=2.589980821 podStartE2EDuration="3.089432846s" totalImagesPullingTime="499.452025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:36:06.088917002 +0000 UTC m=+1953.387394759" watchObservedRunningTime="2026-07-29 21:36:06.089432846 +0000 UTC m=+1953.387910601"
Jul 29 21:36:06.107090 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:06.106576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" podStartSLOduration=2.10655003 podStartE2EDuration="2.10655003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:36:06.104599002 +0000 UTC m=+1953.403076784" watchObservedRunningTime="2026-07-29 21:36:06.10655003 +0000 UTC m=+1953.405027788"
Jul 29 21:36:12.074060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:12.074028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:17.069984 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:17.069939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:17.070411 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:17.070171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:34.541838 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.541802    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:34.542355 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.542083    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" podUID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" containerName="maas-api" containerID="cri-o://28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc" gracePeriod=30
Jul 29 21:36:34.602783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.602742    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:34.603051 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.603022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" podUID="06d0f57a-cfb3-4377-8444-704f8354273c" containerName="payload-processing" containerID="cri-o://be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53" gracePeriod=30
Jul 29 21:36:34.609734 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.609685    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:34.609967 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.609942    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" podUID="96789efd-a432-45d2-ac4f-34275ccb6be7" containerName="payload-pre-processing" containerID="cri-o://223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27" gracePeriod=30
Jul 29 21:36:34.915869 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.915839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:34.919268 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.919238    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:34.926337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume\") pod \"06d0f57a-cfb3-4377-8444-704f8354273c\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") "
Jul 29 21:36:34.926493 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njhtr\" (UniqueName: \"kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr\") pod \"06d0f57a-cfb3-4377-8444-704f8354273c\" (UID: \"06d0f57a-cfb3-4377-8444-704f8354273c\") "
Jul 29 21:36:34.926493 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") pod \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") "
Jul 29 21:36:34.926618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca\") pod \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") "
Jul 29 21:36:34.926618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49xqv\" (UniqueName: \"kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv\") pod \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\" (UID: \"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b\") "
Jul 29 21:36:34.926839 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06d0f57a-cfb3-4377-8444-704f8354273c" (UID: "06d0f57a-cfb3-4377-8444-704f8354273c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:36:34.926948 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" (UID: "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:36:34.926948 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.926938    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d0f57a-cfb3-4377-8444-704f8354273c-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:34.928659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.928635    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr" (OuterVolumeSpecName: "kube-api-access-njhtr") pod "06d0f57a-cfb3-4377-8444-704f8354273c" (UID: "06d0f57a-cfb3-4377-8444-704f8354273c"). InnerVolumeSpecName "kube-api-access-njhtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:36:34.928802 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.928785    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" (UID: "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:36:34.928854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.928815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv" (OuterVolumeSpecName: "kube-api-access-49xqv") pod "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" (UID: "e5618028-a7f4-4cb1-97b8-1888a9bb6a0b"). InnerVolumeSpecName "kube-api-access-49xqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:36:34.933395 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:34.933371    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:35.027974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.027935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume\") pod \"96789efd-a432-45d2-ac4f-34275ccb6be7\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") "
Jul 29 21:36:35.028198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdfkc\" (UniqueName: \"kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc\") pod \"96789efd-a432-45d2-ac4f-34275ccb6be7\" (UID: \"96789efd-a432-45d2-ac4f-34275ccb6be7\") "
Jul 29 21:36:35.028267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028202    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.028267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028213    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49xqv\" (UniqueName: \"kubernetes.io/projected/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-kube-api-access-49xqv\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.028267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028222    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njhtr\" (UniqueName: \"kubernetes.io/projected/06d0f57a-cfb3-4377-8444-704f8354273c-kube-api-access-njhtr\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.028267 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028232    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.028477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.028440    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96789efd-a432-45d2-ac4f-34275ccb6be7" (UID: "96789efd-a432-45d2-ac4f-34275ccb6be7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:36:35.030345 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.030320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc" (OuterVolumeSpecName: "kube-api-access-tdfkc") pod "96789efd-a432-45d2-ac4f-34275ccb6be7" (UID: "96789efd-a432-45d2-ac4f-34275ccb6be7"). InnerVolumeSpecName "kube-api-access-tdfkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:36:35.129262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.129218    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96789efd-a432-45d2-ac4f-34275ccb6be7-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.129262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.129254    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfkc\" (UniqueName: \"kubernetes.io/projected/96789efd-a432-45d2-ac4f-34275ccb6be7-kube-api-access-tdfkc\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:36:35.227404 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.227355    2581 generic.go:356] "Generic (PLEG): container finished" podID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" containerID="28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc" exitCode=0
Jul 29 21:36:35.227616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.227410    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"
Jul 29 21:36:35.227616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.227418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" event={"ID":"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b","Type":"ContainerDied","Data":"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"}
Jul 29 21:36:35.227616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.227468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6" event={"ID":"e5618028-a7f4-4cb1-97b8-1888a9bb6a0b","Type":"ContainerDied","Data":"02296bfd7cba6bf3f7c9a65ed02de4a992b41e0ff39c558c53e7c4dce8d102ca"}
Jul 29 21:36:35.227616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.227498    2581 scope.go:117] "RemoveContainer" containerID="28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"
Jul 29 21:36:35.229104 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.229073    2581 generic.go:356] "Generic (PLEG): container finished" podID="96789efd-a432-45d2-ac4f-34275ccb6be7" containerID="223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27" exitCode=0
Jul 29 21:36:35.229199 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.229115    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"
Jul 29 21:36:35.229441 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.229420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" event={"ID":"96789efd-a432-45d2-ac4f-34275ccb6be7","Type":"ContainerDied","Data":"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"}
Jul 29 21:36:35.229498 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.229477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb" event={"ID":"96789efd-a432-45d2-ac4f-34275ccb6be7","Type":"ContainerDied","Data":"fabd951999e2157c408d89e1d42b5c341942c09ad7f76c928b7dbb3b9297ce73"}
Jul 29 21:36:35.231032 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.231010    2581 generic.go:356] "Generic (PLEG): container finished" podID="06d0f57a-cfb3-4377-8444-704f8354273c" containerID="be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53" exitCode=0
Jul 29 21:36:35.231150 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.231079    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"
Jul 29 21:36:35.231222 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.231082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" event={"ID":"06d0f57a-cfb3-4377-8444-704f8354273c","Type":"ContainerDied","Data":"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"}
Jul 29 21:36:35.231285 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.231254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt" event={"ID":"06d0f57a-cfb3-4377-8444-704f8354273c","Type":"ContainerDied","Data":"8c3e5ceb2994d66f5b712ba998ab7af8f3e1c8b902fdd14d4c42ea6a1f2816a5"}
Jul 29 21:36:35.238461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.238440    2581 scope.go:117] "RemoveContainer" containerID="28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"
Jul 29 21:36:35.238818 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:36:35.238795    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc\": container with ID starting with 28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc not found: ID does not exist" containerID="28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"
Jul 29 21:36:35.238973 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.238828    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc"} err="failed to get container status \"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc\": rpc error: code = NotFound desc = could not find container \"28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc\": container with ID starting with 28cfe7dcc693998aa248cfbb8a78769dc8eadb6ed78b5b6c46839a36be5ec4dc not found: ID does not exist"
Jul 29 21:36:35.238973 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.238851    2581 scope.go:117] "RemoveContainer" containerID="223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"
Jul 29 21:36:35.247783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.247755    2581 scope.go:117] "RemoveContainer" containerID="223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"
Jul 29 21:36:35.248112 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:36:35.248092    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27\": container with ID starting with 223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27 not found: ID does not exist" containerID="223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"
Jul 29 21:36:35.248186 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.248121    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27"} err="failed to get container status \"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27\": rpc error: code = NotFound desc = could not find container \"223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27\": container with ID starting with 223ec34103d8656825c4caf1fe9ce7006298df858f999add0bcc16b156f33d27 not found: ID does not exist"
Jul 29 21:36:35.248186 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.248140    2581 scope.go:117] "RemoveContainer" containerID="be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"
Jul 29 21:36:35.253457 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.253424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:35.257605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.257570    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-592dab8c-6fc8db5f7b-t2jq6"]
Jul 29 21:36:35.258411 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.258389    2581 scope.go:117] "RemoveContainer" containerID="be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"
Jul 29 21:36:35.258964 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:36:35.258933    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53\": container with ID starting with be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53 not found: ID does not exist" containerID="be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"
Jul 29 21:36:35.259052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.258970    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53"} err="failed to get container status \"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53\": rpc error: code = NotFound desc = could not find container \"be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53\": container with ID starting with be6c56d5c6021a18e704f4057c98a9b0f80472a179d0081c42edde10ffe46a53 not found: ID does not exist"
Jul 29 21:36:35.268376 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.268340    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:35.273850 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.273820    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-592dab8c-64c76fcc98-mg2tt"]
Jul 29 21:36:35.284531 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.284496    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:35.288339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:35.288308    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-592dab8c-68d5487fd4-gdhsb"]
Jul 29 21:36:37.187841 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:37.187801    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d0f57a-cfb3-4377-8444-704f8354273c" path="/var/lib/kubelet/pods/06d0f57a-cfb3-4377-8444-704f8354273c/volumes"
Jul 29 21:36:37.188240 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:37.188230    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96789efd-a432-45d2-ac4f-34275ccb6be7" path="/var/lib/kubelet/pods/96789efd-a432-45d2-ac4f-34275ccb6be7/volumes"
Jul 29 21:36:37.188598 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:36:37.188583    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" path="/var/lib/kubelet/pods/e5618028-a7f4-4cb1-97b8-1888a9bb6a0b/volumes"
Jul 29 21:38:22.266103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266013    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266582    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" containerName="maas-api"
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266603    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" containerName="maas-api"
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266621    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d0f57a-cfb3-4377-8444-704f8354273c" containerName="payload-processing"
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266630    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d0f57a-cfb3-4377-8444-704f8354273c" containerName="payload-processing"
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96789efd-a432-45d2-ac4f-34275ccb6be7" containerName="payload-pre-processing"
Jul 29 21:38:22.266687 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266659    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96789efd-a432-45d2-ac4f-34275ccb6be7" containerName="payload-pre-processing"
Jul 29 21:38:22.267198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266837    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96789efd-a432-45d2-ac4f-34275ccb6be7" containerName="payload-pre-processing"
Jul 29 21:38:22.267198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266853    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5618028-a7f4-4cb1-97b8-1888a9bb6a0b" containerName="maas-api"
Jul 29 21:38:22.267198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.266877    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d0f57a-cfb3-4377-8444-704f8354273c" containerName="payload-processing"
Jul 29 21:38:22.269930 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.269909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.273112 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.273094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-095b8200-serving-cert\""
Jul 29 21:38:22.283253 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.283200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:38:22.427060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.427024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.427282 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.427148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pg4\" (UniqueName: \"kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.427282 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.427214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.528369 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.528275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pg4\" (UniqueName: \"kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.528369 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.528322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.528631 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.528391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.528631 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:38:22.528507    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-serving-cert: secret "maas-api-e2e-mt-095b8200-serving-cert" not found
Jul 29 21:38:22.528631 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:38:22.528581    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls podName:51c17be2-a45c-4deb-8f03-bd5f73135ea0 nodeName:}" failed. No retries permitted until 2026-07-29 21:38:23.028557495 +0000 UTC m=+2090.327035245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls") pod "maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" (UID: "51c17be2-a45c-4deb-8f03-bd5f73135ea0") : secret "maas-api-e2e-mt-095b8200-serving-cert" not found
Jul 29 21:38:22.529138 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.529116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:22.544236 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:22.544199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pg4\" (UniqueName: \"kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:23.034761 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.034683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:23.038169 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.038133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") pod \"maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:23.182743 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.182670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:23.229955 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.229917    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:38:23.233744 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.233716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.237052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.237020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-095b8200\""
Jul 29 21:38:23.238005 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.237930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-095b8200-dockercfg-5m99g\""
Jul 29 21:38:23.248320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.248125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:38:23.276570 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.276506    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:38:23.280409 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.280321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.306899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.306864    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:38:23.337572 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.337513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlpp\" (UniqueName: \"kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.337833 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.337618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.353587 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.353424    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:38:23.357553 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:38:23.357523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c17be2_a45c_4deb_8f03_bd5f73135ea0.slice/crio-f3b97e6a9734029853dfc699f76c1a81ca7027a4348118b937525aa8bf11a1ab WatchSource:0}: Error finding container f3b97e6a9734029853dfc699f76c1a81ca7027a4348118b937525aa8bf11a1ab: Status 404 returned error can't find the container with id f3b97e6a9734029853dfc699f76c1a81ca7027a4348118b937525aa8bf11a1ab
Jul 29 21:38:23.439211 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.439167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlpp\" (UniqueName: \"kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.439399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.439218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.439399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.439254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrtw\" (UniqueName: \"kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.439399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.439301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.439966 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.439946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.448535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.448499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlpp\" (UniqueName: \"kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp\") pod \"payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") " pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.539948 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.539908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.540142 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.539959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrtw\" (UniqueName: \"kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.540642 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.540622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.549447 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.549419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.550550 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.550527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrtw\" (UniqueName: \"kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw\") pod \"payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.614057 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.614028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:23.711950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.711914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:38:23.714269 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:38:23.714239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec170119_c659_45bb_ae88_dc06b108d41b.slice/crio-6d710049236d30f64dde2d9fe5c9b78665a6c92fdda8caf32b0b3e6dfcb9d178 WatchSource:0}: Error finding container 6d710049236d30f64dde2d9fe5c9b78665a6c92fdda8caf32b0b3e6dfcb9d178: Status 404 returned error can't find the container with id 6d710049236d30f64dde2d9fe5c9b78665a6c92fdda8caf32b0b3e6dfcb9d178
Jul 29 21:38:23.780017 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.779772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:38:23.782835 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:38:23.782802    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398e460f_fe0e_4434_8361_f423b44e7fce.slice/crio-488cad2eab7dd508d35096191e3b8f1dd51a41c6daf08722432a39df0f643de0 WatchSource:0}: Error finding container 488cad2eab7dd508d35096191e3b8f1dd51a41c6daf08722432a39df0f643de0: Status 404 returned error can't find the container with id 488cad2eab7dd508d35096191e3b8f1dd51a41c6daf08722432a39df0f643de0
Jul 29 21:38:23.837779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.837744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" event={"ID":"398e460f-fe0e-4434-8361-f423b44e7fce","Type":"ContainerStarted","Data":"488cad2eab7dd508d35096191e3b8f1dd51a41c6daf08722432a39df0f643de0"}
Jul 29 21:38:23.839897 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.839865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" event={"ID":"ec170119-c659-45bb-ae88-dc06b108d41b","Type":"ContainerStarted","Data":"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"}
Jul 29 21:38:23.840061 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.839929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" event={"ID":"ec170119-c659-45bb-ae88-dc06b108d41b","Type":"ContainerStarted","Data":"6d710049236d30f64dde2d9fe5c9b78665a6c92fdda8caf32b0b3e6dfcb9d178"}
Jul 29 21:38:23.840061 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.840027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:23.841403 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.841378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" event={"ID":"51c17be2-a45c-4deb-8f03-bd5f73135ea0","Type":"ContainerStarted","Data":"f3b97e6a9734029853dfc699f76c1a81ca7027a4348118b937525aa8bf11a1ab"}
Jul 29 21:38:23.857747 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:23.857681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" podStartSLOduration=0.857664901 podStartE2EDuration="857.664901ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:38:23.857122237 +0000 UTC m=+2091.155599994" watchObservedRunningTime="2026-07-29 21:38:23.857664901 +0000 UTC m=+2091.156142655"
Jul 29 21:38:24.849625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.849584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" event={"ID":"398e460f-fe0e-4434-8361-f423b44e7fce","Type":"ContainerStarted","Data":"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"}
Jul 29 21:38:24.850147 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.849674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:38:24.851352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.851328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" event={"ID":"51c17be2-a45c-4deb-8f03-bd5f73135ea0","Type":"ContainerStarted","Data":"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"}
Jul 29 21:38:24.851542 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.851530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:24.870238 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.870187    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" podStartSLOduration=1.870169686 podStartE2EDuration="1.870169686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:38:24.868641228 +0000 UTC m=+2092.167118984" watchObservedRunningTime="2026-07-29 21:38:24.870169686 +0000 UTC m=+2092.168647473"
Jul 29 21:38:24.889894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:24.889834    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" podStartSLOduration=2.4363749820000002 podStartE2EDuration="2.889813937s" totalImagesPullingTime="453.438955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:38:24.888003541 +0000 UTC m=+2092.186481297" watchObservedRunningTime="2026-07-29 21:38:24.889813937 +0000 UTC m=+2092.188291691"
Jul 29 21:38:30.863216 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:30.863180    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:38:33.271377 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:33.271336    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:38:33.277372 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:33.277347    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:38:34.853779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:34.853745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:38:35.859896 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:38:35.859861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:39:05.057432 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.057383    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:39:05.058091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.057642    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" podUID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" containerName="maas-api" containerID="cri-o://3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82" gracePeriod=30
Jul 29 21:39:05.168292 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.168257    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:39:05.168549 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.168517    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" podUID="ec170119-c659-45bb-ae88-dc06b108d41b" containerName="payload-processing" containerID="cri-o://ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59" gracePeriod=30
Jul 29 21:39:05.203364 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.202414    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:39:05.203364 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.202663    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" podUID="398e460f-fe0e-4434-8361-f423b44e7fce" containerName="payload-pre-processing" containerID="cri-o://089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7" gracePeriod=30
Jul 29 21:39:05.350476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.350448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:39:05.399355 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.399279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca\") pod \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") "
Jul 29 21:39:05.399655 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.399434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") pod \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") "
Jul 29 21:39:05.399655 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.399520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6pg4\" (UniqueName: \"kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4\") pod \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\" (UID: \"51c17be2-a45c-4deb-8f03-bd5f73135ea0\") "
Jul 29 21:39:05.399849 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.399686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51c17be2-a45c-4deb-8f03-bd5f73135ea0" (UID: "51c17be2-a45c-4deb-8f03-bd5f73135ea0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:39:05.400567 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.400539    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c17be2-a45c-4deb-8f03-bd5f73135ea0-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.403559 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.403520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4" (OuterVolumeSpecName: "kube-api-access-x6pg4") pod "51c17be2-a45c-4deb-8f03-bd5f73135ea0" (UID: "51c17be2-a45c-4deb-8f03-bd5f73135ea0"). InnerVolumeSpecName "kube-api-access-x6pg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:39:05.403559 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.403529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51c17be2-a45c-4deb-8f03-bd5f73135ea0" (UID: "51c17be2-a45c-4deb-8f03-bd5f73135ea0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:39:05.472718 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.472669    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:39:05.482134 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.482113    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:39:05.501380 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.501352    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c17be2-a45c-4deb-8f03-bd5f73135ea0-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.501380 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.501378    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6pg4\" (UniqueName: \"kubernetes.io/projected/51c17be2-a45c-4deb-8f03-bd5f73135ea0-kube-api-access-x6pg4\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.602243 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume\") pod \"ec170119-c659-45bb-ae88-dc06b108d41b\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") "
Jul 29 21:39:05.602437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stlpp\" (UniqueName: \"kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp\") pod \"ec170119-c659-45bb-ae88-dc06b108d41b\" (UID: \"ec170119-c659-45bb-ae88-dc06b108d41b\") "
Jul 29 21:39:05.602437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602287    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qrtw\" (UniqueName: \"kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw\") pod \"398e460f-fe0e-4434-8361-f423b44e7fce\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") "
Jul 29 21:39:05.602437 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume\") pod \"398e460f-fe0e-4434-8361-f423b44e7fce\" (UID: \"398e460f-fe0e-4434-8361-f423b44e7fce\") "
Jul 29 21:39:05.602825 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec170119-c659-45bb-ae88-dc06b108d41b" (UID: "ec170119-c659-45bb-ae88-dc06b108d41b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:39:05.602825 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.602805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "398e460f-fe0e-4434-8361-f423b44e7fce" (UID: "398e460f-fe0e-4434-8361-f423b44e7fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:39:05.604594 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.604567    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp" (OuterVolumeSpecName: "kube-api-access-stlpp") pod "ec170119-c659-45bb-ae88-dc06b108d41b" (UID: "ec170119-c659-45bb-ae88-dc06b108d41b"). InnerVolumeSpecName "kube-api-access-stlpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:39:05.604678 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.604628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw" (OuterVolumeSpecName: "kube-api-access-6qrtw") pod "398e460f-fe0e-4434-8361-f423b44e7fce" (UID: "398e460f-fe0e-4434-8361-f423b44e7fce"). InnerVolumeSpecName "kube-api-access-6qrtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:39:05.704011 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.703970    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec170119-c659-45bb-ae88-dc06b108d41b-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.704011 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.704002    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlpp\" (UniqueName: \"kubernetes.io/projected/ec170119-c659-45bb-ae88-dc06b108d41b-kube-api-access-stlpp\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.704236 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.704013    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrtw\" (UniqueName: \"kubernetes.io/projected/398e460f-fe0e-4434-8361-f423b44e7fce-kube-api-access-6qrtw\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:05.704236 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:05.704083    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398e460f-fe0e-4434-8361-f423b44e7fce-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:39:06.083937 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.083904    2581 generic.go:356] "Generic (PLEG): container finished" podID="ec170119-c659-45bb-ae88-dc06b108d41b" containerID="ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59" exitCode=0
Jul 29 21:39:06.084397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.083955    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"
Jul 29 21:39:06.084397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.083987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" event={"ID":"ec170119-c659-45bb-ae88-dc06b108d41b","Type":"ContainerDied","Data":"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"}
Jul 29 21:39:06.084397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.084056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82" event={"ID":"ec170119-c659-45bb-ae88-dc06b108d41b","Type":"ContainerDied","Data":"6d710049236d30f64dde2d9fe5c9b78665a6c92fdda8caf32b0b3e6dfcb9d178"}
Jul 29 21:39:06.084397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.084095    2581 scope.go:117] "RemoveContainer" containerID="ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.085378    2581 generic.go:356] "Generic (PLEG): container finished" podID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" containerID="3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82" exitCode=0
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.085417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.085503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" event={"ID":"51c17be2-a45c-4deb-8f03-bd5f73135ea0","Type":"ContainerDied","Data":"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"}
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.085546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8" event={"ID":"51c17be2-a45c-4deb-8f03-bd5f73135ea0","Type":"ContainerDied","Data":"f3b97e6a9734029853dfc699f76c1a81ca7027a4348118b937525aa8bf11a1ab"}
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.087083    2581 generic.go:356] "Generic (PLEG): container finished" podID="398e460f-fe0e-4434-8361-f423b44e7fce" containerID="089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7" exitCode=0
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.087111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" event={"ID":"398e460f-fe0e-4434-8361-f423b44e7fce","Type":"ContainerDied","Data":"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"}
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.087151    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"
Jul 29 21:39:06.087888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.087187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn" event={"ID":"398e460f-fe0e-4434-8361-f423b44e7fce","Type":"ContainerDied","Data":"488cad2eab7dd508d35096191e3b8f1dd51a41c6daf08722432a39df0f643de0"}
Jul 29 21:39:06.097331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.097310    2581 scope.go:117] "RemoveContainer" containerID="ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"
Jul 29 21:39:06.097841 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:39:06.097815    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59\": container with ID starting with ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59 not found: ID does not exist" containerID="ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"
Jul 29 21:39:06.097979 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.097855    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59"} err="failed to get container status \"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59\": rpc error: code = NotFound desc = could not find container \"ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59\": container with ID starting with ac34a4910a29688f801f84acefdb3cc4065a7206b5ca89240dba3cbc2b9e3c59 not found: ID does not exist"
Jul 29 21:39:06.097979 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.097882    2581 scope.go:117] "RemoveContainer" containerID="3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"
Jul 29 21:39:06.107947 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.107928    2581 scope.go:117] "RemoveContainer" containerID="3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"
Jul 29 21:39:06.108267 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:39:06.108245    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82\": container with ID starting with 3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82 not found: ID does not exist" containerID="3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"
Jul 29 21:39:06.108337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.108275    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82"} err="failed to get container status \"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82\": rpc error: code = NotFound desc = could not find container \"3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82\": container with ID starting with 3eb6eb33bd5116e9a86dc0a6297d6a97acd12643e1ca56053c88e47b3ae61e82 not found: ID does not exist"
Jul 29 21:39:06.108337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.108295    2581 scope.go:117] "RemoveContainer" containerID="089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"
Jul 29 21:39:06.114387 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.114356    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:39:06.118192 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.118171    2581 scope.go:117] "RemoveContainer" containerID="089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"
Jul 29 21:39:06.118537 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:39:06.118517    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7\": container with ID starting with 089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7 not found: ID does not exist" containerID="089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"
Jul 29 21:39:06.118598 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.118547    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7"} err="failed to get container status \"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7\": rpc error: code = NotFound desc = could not find container \"089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7\": container with ID starting with 089094353d5ee5e6365da21f87d9cd6ce65b588173d28b6634197ad600117cd7 not found: ID does not exist"
Jul 29 21:39:06.118735 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.118686    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-095b8200-5fff5b8786-8tv82"]
Jul 29 21:39:06.129963 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.129932    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:39:06.134094 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.134066    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-095b8200-58c8d6cc5d-wsqg8"]
Jul 29 21:39:06.145455 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.145428    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:39:06.150077 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:06.150050    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-095b8200-586d5697b4-drmkn"]
Jul 29 21:39:07.187408 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:07.187373    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398e460f-fe0e-4434-8361-f423b44e7fce" path="/var/lib/kubelet/pods/398e460f-fe0e-4434-8361-f423b44e7fce/volumes"
Jul 29 21:39:07.187820 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:07.187786    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" path="/var/lib/kubelet/pods/51c17be2-a45c-4deb-8f03-bd5f73135ea0/volumes"
Jul 29 21:39:07.188138 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:07.188122    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec170119-c659-45bb-ae88-dc06b108d41b" path="/var/lib/kubelet/pods/ec170119-c659-45bb-ae88-dc06b108d41b/volumes"
Jul 29 21:39:36.576595 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.576557    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577009    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398e460f-fe0e-4434-8361-f423b44e7fce" containerName="payload-pre-processing"
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577022    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="398e460f-fe0e-4434-8361-f423b44e7fce" containerName="payload-pre-processing"
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577035    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" containerName="maas-api"
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577041    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" containerName="maas-api"
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec170119-c659-45bb-ae88-dc06b108d41b" containerName="payload-processing"
Jul 29 21:39:36.577101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec170119-c659-45bb-ae88-dc06b108d41b" containerName="payload-processing"
Jul 29 21:39:36.577298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577136    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c17be2-a45c-4deb-8f03-bd5f73135ea0" containerName="maas-api"
Jul 29 21:39:36.577298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577144    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="398e460f-fe0e-4434-8361-f423b44e7fce" containerName="payload-pre-processing"
Jul 29 21:39:36.577298 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.577153    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec170119-c659-45bb-ae88-dc06b108d41b" containerName="payload-processing"
Jul 29 21:39:36.580189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.580164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.582633 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.582601    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a6a723f4-serving-cert\""
Jul 29 21:39:36.590945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.590659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.590945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.590747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dkz\" (UniqueName: \"kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.590945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.590807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.591235 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.591000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:39:36.691525 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.691483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.691782 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.691532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dkz\" (UniqueName: \"kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.691782 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.691591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.691932 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:39:36.691803    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-serving-cert: secret "maas-api-e2e-mt-a6a723f4-serving-cert" not found
Jul 29 21:39:36.691932 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:39:36.691872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls podName:008f7561-d2fc-442c-bbf9-be76c9908a1a nodeName:}" failed. No retries permitted until 2026-07-29 21:39:37.191851236 +0000 UTC m=+2164.490328976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls") pod "maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" (UID: "008f7561-d2fc-442c-bbf9-be76c9908a1a") : secret "maas-api-e2e-mt-a6a723f4-serving-cert" not found
Jul 29 21:39:36.692397 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.692371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:36.704379 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:36.704347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dkz\" (UniqueName: \"kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:37.200903 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.200859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:37.203456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.203435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") pod \"maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:37.214605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.214565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:39:37.218347 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.218324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.222109 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.222083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a6a723f4\""
Jul 29 21:39:37.222109 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.222101    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a6a723f4-dockercfg-ndhrv\""
Jul 29 21:39:37.229357 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.229326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:39:37.244514 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.244480    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:39:37.247995 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.247968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.260322 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.260296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:39:37.302006 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.301964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6tk\" (UniqueName: \"kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.302207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.302099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.302207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.302151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.302207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.302171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrv2\" (UniqueName: \"kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.402929 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.402890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.403122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.402941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.403122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.402967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrv2\" (UniqueName: \"kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.403122 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.403029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6tk\" (UniqueName: \"kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.403785 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.403761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.403969 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.403950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.412770 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.412741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6tk\" (UniqueName: \"kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk\") pod \"payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") " pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.413592 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.413568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrv2\" (UniqueName: \"kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2\") pod \"payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.493583 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.493482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:37.531126 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.531091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:37.559035 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.559000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:37.684862 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.684779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:39:37.687521 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:39:37.687429    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008f7561_d2fc_442c_bbf9_be76c9908a1a.slice/crio-fd656b54b7c4af5c89a712b534e7ca40d2af0f144a1f3f805eb353bbe6ca1fd2 WatchSource:0}: Error finding container fd656b54b7c4af5c89a712b534e7ca40d2af0f144a1f3f805eb353bbe6ca1fd2: Status 404 returned error can't find the container with id fd656b54b7c4af5c89a712b534e7ca40d2af0f144a1f3f805eb353bbe6ca1fd2
Jul 29 21:39:37.689533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.689501    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:39:37.744598 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.744369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:39:37.749296 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:39:37.749267    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8feb4e_9bb5_4724_80b6_951d2a9d33a4.slice/crio-afbac6f12ae11f6fae835fcea27465b3e04811cf8db2ba35f2b55d3bc2dbb561 WatchSource:0}: Error finding container afbac6f12ae11f6fae835fcea27465b3e04811cf8db2ba35f2b55d3bc2dbb561: Status 404 returned error can't find the container with id afbac6f12ae11f6fae835fcea27465b3e04811cf8db2ba35f2b55d3bc2dbb561
Jul 29 21:39:37.774013 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:37.773986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:39:37.776159 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:39:37.776126    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c55fd3a_86bd_42a5_a507_1177fd161707.slice/crio-4fdec83a2375ae3014a2f63691eb4f6b54875ded9e8d1835f22146699b01aa5f WatchSource:0}: Error finding container 4fdec83a2375ae3014a2f63691eb4f6b54875ded9e8d1835f22146699b01aa5f: Status 404 returned error can't find the container with id 4fdec83a2375ae3014a2f63691eb4f6b54875ded9e8d1835f22146699b01aa5f
Jul 29 21:39:38.272891 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.272853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" event={"ID":"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4","Type":"ContainerStarted","Data":"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"}
Jul 29 21:39:38.273068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.272919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" event={"ID":"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4","Type":"ContainerStarted","Data":"afbac6f12ae11f6fae835fcea27465b3e04811cf8db2ba35f2b55d3bc2dbb561"}
Jul 29 21:39:38.273068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.272961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:38.274887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.274832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" event={"ID":"2c55fd3a-86bd-42a5-a507-1177fd161707","Type":"ContainerStarted","Data":"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"}
Jul 29 21:39:38.274887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.274888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:39:38.275121 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.274911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" event={"ID":"2c55fd3a-86bd-42a5-a507-1177fd161707","Type":"ContainerStarted","Data":"4fdec83a2375ae3014a2f63691eb4f6b54875ded9e8d1835f22146699b01aa5f"}
Jul 29 21:39:38.276879 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.276852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" event={"ID":"008f7561-d2fc-442c-bbf9-be76c9908a1a","Type":"ContainerStarted","Data":"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"}
Jul 29 21:39:38.277016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.276902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" event={"ID":"008f7561-d2fc-442c-bbf9-be76c9908a1a","Type":"ContainerStarted","Data":"fd656b54b7c4af5c89a712b534e7ca40d2af0f144a1f3f805eb353bbe6ca1fd2"}
Jul 29 21:39:38.277016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.277008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:38.290855 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.290800    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" podStartSLOduration=1.290757337 podStartE2EDuration="1.290757337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:39:38.290095727 +0000 UTC m=+2165.588573485" watchObservedRunningTime="2026-07-29 21:39:38.290757337 +0000 UTC m=+2165.589235092"
Jul 29 21:39:38.306973 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.306908    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" podStartSLOduration=1.306887841 podStartE2EDuration="1.306887841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:39:38.305898804 +0000 UTC m=+2165.604376559" watchObservedRunningTime="2026-07-29 21:39:38.306887841 +0000 UTC m=+2165.605365595"
Jul 29 21:39:38.325060 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:38.324952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" podStartSLOduration=1.9551403299999999 podStartE2EDuration="2.324932474s" totalImagesPullingTime="369.792144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:39:38.321977376 +0000 UTC m=+2165.620455132" watchObservedRunningTime="2026-07-29 21:39:38.324932474 +0000 UTC m=+2165.623410229"
Jul 29 21:39:44.288535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:44.288504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:39:49.285665 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:49.285634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:39:49.286103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:39:49.285722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:40:45.693737 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.693672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:40:45.694630 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.693921    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" podUID="008f7561-d2fc-442c-bbf9-be76c9908a1a" containerName="maas-api" containerID="cri-o://5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47" gracePeriod=30
Jul 29 21:40:45.739090 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.738989    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:40:45.739467 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.739429    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" podUID="2c55fd3a-86bd-42a5-a507-1177fd161707" containerName="payload-processing" containerID="cri-o://e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64" gracePeriod=30
Jul 29 21:40:45.746356 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.746284    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:40:45.746556 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:45.746528    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" podUID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" containerName="payload-pre-processing" containerID="cri-o://0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e" gracePeriod=30
Jul 29 21:40:46.022951 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.022909    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:40:46.026067 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.026040    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:40:46.055354 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.055322    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:40:46.134118 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") pod \"008f7561-d2fc-442c-bbf9-be76c9908a1a\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") "
Jul 29 21:40:46.134118 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume\") pod \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") "
Jul 29 21:40:46.134375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2dkz\" (UniqueName: \"kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz\") pod \"008f7561-d2fc-442c-bbf9-be76c9908a1a\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") "
Jul 29 21:40:46.134375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddrv2\" (UniqueName: \"kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2\") pod \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\" (UID: \"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4\") "
Jul 29 21:40:46.134375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q6tk\" (UniqueName: \"kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk\") pod \"2c55fd3a-86bd-42a5-a507-1177fd161707\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") "
Jul 29 21:40:46.134375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume\") pod \"2c55fd3a-86bd-42a5-a507-1177fd161707\" (UID: \"2c55fd3a-86bd-42a5-a507-1177fd161707\") "
Jul 29 21:40:46.134375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca\") pod \"008f7561-d2fc-442c-bbf9-be76c9908a1a\" (UID: \"008f7561-d2fc-442c-bbf9-be76c9908a1a\") "
Jul 29 21:40:46.134669 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" (UID: "4f8feb4e-9bb5-4724-80b6-951d2a9d33a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:40:46.134949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c55fd3a-86bd-42a5-a507-1177fd161707" (UID: "2c55fd3a-86bd-42a5-a507-1177fd161707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:40:46.134949 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.134910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "008f7561-d2fc-442c-bbf9-be76c9908a1a" (UID: "008f7561-d2fc-442c-bbf9-be76c9908a1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:40:46.136579 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.136550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "008f7561-d2fc-442c-bbf9-be76c9908a1a" (UID: "008f7561-d2fc-442c-bbf9-be76c9908a1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:40:46.136671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.136595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk" (OuterVolumeSpecName: "kube-api-access-4q6tk") pod "2c55fd3a-86bd-42a5-a507-1177fd161707" (UID: "2c55fd3a-86bd-42a5-a507-1177fd161707"). InnerVolumeSpecName "kube-api-access-4q6tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:40:46.136671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.136636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2" (OuterVolumeSpecName: "kube-api-access-ddrv2") pod "4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" (UID: "4f8feb4e-9bb5-4724-80b6-951d2a9d33a4"). InnerVolumeSpecName "kube-api-access-ddrv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:40:46.136671 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.136661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz" (OuterVolumeSpecName: "kube-api-access-j2dkz") pod "008f7561-d2fc-442c-bbf9-be76c9908a1a" (UID: "008f7561-d2fc-442c-bbf9-be76c9908a1a"). InnerVolumeSpecName "kube-api-access-j2dkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235849    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6tk\" (UniqueName: \"kubernetes.io/projected/2c55fd3a-86bd-42a5-a507-1177fd161707-kube-api-access-4q6tk\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235879    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55fd3a-86bd-42a5-a507-1177fd161707-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235888    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7561-d2fc-442c-bbf9-be76c9908a1a-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235898    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008f7561-d2fc-442c-bbf9-be76c9908a1a-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235908    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235916    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dkz\" (UniqueName: \"kubernetes.io/projected/008f7561-d2fc-442c-bbf9-be76c9908a1a-kube-api-access-j2dkz\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.235939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.235925    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrv2\" (UniqueName: \"kubernetes.io/projected/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4-kube-api-access-ddrv2\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:40:46.657420 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.657378    2581 generic.go:356] "Generic (PLEG): container finished" podID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" containerID="0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e" exitCode=0
Jul 29 21:40:46.657627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.657427    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"
Jul 29 21:40:46.657627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.657479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" event={"ID":"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4","Type":"ContainerDied","Data":"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"}
Jul 29 21:40:46.657627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.657540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w" event={"ID":"4f8feb4e-9bb5-4724-80b6-951d2a9d33a4","Type":"ContainerDied","Data":"afbac6f12ae11f6fae835fcea27465b3e04811cf8db2ba35f2b55d3bc2dbb561"}
Jul 29 21:40:46.657627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.657620    2581 scope.go:117] "RemoveContainer" containerID="0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"
Jul 29 21:40:46.659598 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.659570    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c55fd3a-86bd-42a5-a507-1177fd161707" containerID="e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64" exitCode=0
Jul 29 21:40:46.659815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.659614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"
Jul 29 21:40:46.659815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.659732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" event={"ID":"2c55fd3a-86bd-42a5-a507-1177fd161707","Type":"ContainerDied","Data":"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"}
Jul 29 21:40:46.659815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.659801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2" event={"ID":"2c55fd3a-86bd-42a5-a507-1177fd161707","Type":"ContainerDied","Data":"4fdec83a2375ae3014a2f63691eb4f6b54875ded9e8d1835f22146699b01aa5f"}
Jul 29 21:40:46.661783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.661677    2581 generic.go:356] "Generic (PLEG): container finished" podID="008f7561-d2fc-442c-bbf9-be76c9908a1a" containerID="5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47" exitCode=0
Jul 29 21:40:46.661783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.661749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"
Jul 29 21:40:46.661992 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.661742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" event={"ID":"008f7561-d2fc-442c-bbf9-be76c9908a1a","Type":"ContainerDied","Data":"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"}
Jul 29 21:40:46.661992 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.661884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc" event={"ID":"008f7561-d2fc-442c-bbf9-be76c9908a1a","Type":"ContainerDied","Data":"fd656b54b7c4af5c89a712b534e7ca40d2af0f144a1f3f805eb353bbe6ca1fd2"}
Jul 29 21:40:46.671779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.671603    2581 scope.go:117] "RemoveContainer" containerID="0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"
Jul 29 21:40:46.671968 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:40:46.671938    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e\": container with ID starting with 0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e not found: ID does not exist" containerID="0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"
Jul 29 21:40:46.672082 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.671978    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e"} err="failed to get container status \"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e\": rpc error: code = NotFound desc = could not find container \"0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e\": container with ID starting with 0d01480c2f5f051f181df054a012f96dd4f92f012fc6da4a773c529c90f2288e not found: ID does not exist"
Jul 29 21:40:46.672082 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.671995    2581 scope.go:117] "RemoveContainer" containerID="e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"
Jul 29 21:40:46.682570 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.682551    2581 scope.go:117] "RemoveContainer" containerID="e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"
Jul 29 21:40:46.682951 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:40:46.682926    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64\": container with ID starting with e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64 not found: ID does not exist" containerID="e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"
Jul 29 21:40:46.683049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.682962    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64"} err="failed to get container status \"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64\": rpc error: code = NotFound desc = could not find container \"e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64\": container with ID starting with e16315778476d28b9980639042aa43a2fcc01f082779a70c90da862ee0c40d64 not found: ID does not exist"
Jul 29 21:40:46.683049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.682988    2581 scope.go:117] "RemoveContainer" containerID="5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"
Jul 29 21:40:46.685105 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.685085    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:40:46.692427 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.692404    2581 scope.go:117] "RemoveContainer" containerID="5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"
Jul 29 21:40:46.692762 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.692740    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a6a723f4-dd48d8fbd-xzf8w"]
Jul 29 21:40:46.692762 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:40:46.692746    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47\": container with ID starting with 5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47 not found: ID does not exist" containerID="5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"
Jul 29 21:40:46.692886 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.692785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47"} err="failed to get container status \"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47\": rpc error: code = NotFound desc = could not find container \"5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47\": container with ID starting with 5d39af014d9b1fd6c4d899966aeb4afa363a8d98fc759f894bb8438a0ab50c47 not found: ID does not exist"
Jul 29 21:40:46.703517 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.703482    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:40:46.711621 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.711597    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a6a723f4-89d4f8cc8-s8vq2"]
Jul 29 21:40:46.726297 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.726262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:40:46.736388 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:46.736361    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6a723f4-89d6bf794-bsxkc"]
Jul 29 21:40:47.188204 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:47.188171    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="008f7561-d2fc-442c-bbf9-be76c9908a1a" path="/var/lib/kubelet/pods/008f7561-d2fc-442c-bbf9-be76c9908a1a/volumes"
Jul 29 21:40:47.188573 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:47.188559    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c55fd3a-86bd-42a5-a507-1177fd161707" path="/var/lib/kubelet/pods/2c55fd3a-86bd-42a5-a507-1177fd161707/volumes"
Jul 29 21:40:47.188901 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:40:47.188888    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" path="/var/lib/kubelet/pods/4f8feb4e-9bb5-4724-80b6-951d2a9d33a4/volumes"
Jul 29 21:43:02.931893 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.931849    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c55fd3a-86bd-42a5-a507-1177fd161707" containerName="payload-processing"
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c55fd3a-86bd-42a5-a507-1177fd161707" containerName="payload-processing"
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932467    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008f7561-d2fc-442c-bbf9-be76c9908a1a" containerName="maas-api"
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932477    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="008f7561-d2fc-442c-bbf9-be76c9908a1a" containerName="maas-api"
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" containerName="payload-pre-processing"
Jul 29 21:43:02.932529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932513    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" containerName="payload-pre-processing"
Jul 29 21:43:02.932883 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8feb4e-9bb5-4724-80b6-951d2a9d33a4" containerName="payload-pre-processing"
Jul 29 21:43:02.932883 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932630    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="008f7561-d2fc-442c-bbf9-be76c9908a1a" containerName="maas-api"
Jul 29 21:43:02.932883 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.932640    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c55fd3a-86bd-42a5-a507-1177fd161707" containerName="payload-processing"
Jul 29 21:43:02.936260 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.936234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:02.939079 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.939044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-69c0f1-serving-cert\""
Jul 29 21:43:02.947000 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.946965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:43:02.970130 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.970095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:02.970130 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.970145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:02.970371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:02.970169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlf97\" (UniqueName: \"kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.071352 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.071321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.071555 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.071366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.071555 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.071388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlf97\" (UniqueName: \"kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.071555 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:43:03.071493    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-serving-cert: secret "maas-api-e2e-inf-a-69c0f1-serving-cert" not found
Jul 29 21:43:03.071794 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:43:03.071570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls podName:6c8f491f-89b1-4376-8701-a6e54f0aa1be nodeName:}" failed. No retries permitted until 2026-07-29 21:43:03.571552506 +0000 UTC m=+2370.870030241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls") pod "maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" (UID: "6c8f491f-89b1-4376-8701-a6e54f0aa1be") : secret "maas-api-e2e-inf-a-69c0f1-serving-cert" not found
Jul 29 21:43:03.072113 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.072087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.080473 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.080444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlf97\" (UniqueName: \"kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.575944 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.575909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.578538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.578516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.685457 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.685420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:43:03.689037 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.689018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.692472 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.692452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-69c0f1\""
Jul 29 21:43:03.704038 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.704004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:43:03.735016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.734980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:43:03.738365 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.738347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.742015 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.741993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-69c0f1-dockercfg-glhx7\""
Jul 29 21:43:03.762293 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.762255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:43:03.778384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.778348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.778384 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.778383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtnz\" (UniqueName: \"kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.778588 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.778407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxnb\" (UniqueName: \"kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.778588 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.778565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.849167 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.849132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:03.879864 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.879833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.879864 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.879866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtnz\" (UniqueName: \"kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.880099 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.879895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxnb\" (UniqueName: \"kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.880099 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.879998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.880830 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.880801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:03.881086 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.881057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.889448 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.889414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxnb\" (UniqueName: \"kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb\") pod \"payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:03.889594 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:03.889534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtnz\" (UniqueName: \"kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz\") pod \"payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:04.008061 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.008007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:04.010546 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.010522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:43:04.011687 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:04.011655    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8f491f_89b1_4376_8701_a6e54f0aa1be.slice/crio-2b0143134675c797717111fcea0faee344d2c560c0c8ee047d6678ae85f72e6a WatchSource:0}: Error finding container 2b0143134675c797717111fcea0faee344d2c560c0c8ee047d6678ae85f72e6a: Status 404 returned error can't find the container with id 2b0143134675c797717111fcea0faee344d2c560c0c8ee047d6678ae85f72e6a
Jul 29 21:43:04.048605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.048575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:04.170110 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.170077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:43:04.172104 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:04.172071    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646a607f_6df2_4a34_a367_1147adb11d52.slice/crio-bff7c2816240112ca9707279da3b1cd80f9181639cf1f049d2b30b871bd8063b WatchSource:0}: Error finding container bff7c2816240112ca9707279da3b1cd80f9181639cf1f049d2b30b871bd8063b: Status 404 returned error can't find the container with id bff7c2816240112ca9707279da3b1cd80f9181639cf1f049d2b30b871bd8063b
Jul 29 21:43:04.212766 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.212734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:43:04.219265 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:04.219233    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2716e28_b2f2_4885_ad5b_8770aa60b834.slice/crio-af57161e3b9a72195057315328ac956152d1d0561c715706ad54b3ed06028dd9 WatchSource:0}: Error finding container af57161e3b9a72195057315328ac956152d1d0561c715706ad54b3ed06028dd9: Status 404 returned error can't find the container with id af57161e3b9a72195057315328ac956152d1d0561c715706ad54b3ed06028dd9
Jul 29 21:43:04.437271 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.437151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" event={"ID":"6c8f491f-89b1-4376-8701-a6e54f0aa1be","Type":"ContainerStarted","Data":"2b0143134675c797717111fcea0faee344d2c560c0c8ee047d6678ae85f72e6a"}
Jul 29 21:43:04.439263 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.439234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" event={"ID":"f2716e28-b2f2-4885-ad5b-8770aa60b834","Type":"ContainerStarted","Data":"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"}
Jul 29 21:43:04.439433 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.439287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" event={"ID":"f2716e28-b2f2-4885-ad5b-8770aa60b834","Type":"ContainerStarted","Data":"af57161e3b9a72195057315328ac956152d1d0561c715706ad54b3ed06028dd9"}
Jul 29 21:43:04.439610 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.439545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:04.440894 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.440874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" event={"ID":"646a607f-6df2-4a34-a367-1147adb11d52","Type":"ContainerStarted","Data":"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"}
Jul 29 21:43:04.440997 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.440916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" event={"ID":"646a607f-6df2-4a34-a367-1147adb11d52","Type":"ContainerStarted","Data":"bff7c2816240112ca9707279da3b1cd80f9181639cf1f049d2b30b871bd8063b"}
Jul 29 21:43:04.440997 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.440978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:04.459822 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.459756    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" podStartSLOduration=1.45973203 podStartE2EDuration="1.45973203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:04.458368024 +0000 UTC m=+2371.756845780" watchObservedRunningTime="2026-07-29 21:43:04.45973203 +0000 UTC m=+2371.758209763"
Jul 29 21:43:04.477477 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:04.477420    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" podStartSLOduration=1.477400382 podStartE2EDuration="1.477400382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:04.475278425 +0000 UTC m=+2371.773756182" watchObservedRunningTime="2026-07-29 21:43:04.477400382 +0000 UTC m=+2371.775878141"
Jul 29 21:43:05.450004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:05.449953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" event={"ID":"6c8f491f-89b1-4376-8701-a6e54f0aa1be","Type":"ContainerStarted","Data":"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"}
Jul 29 21:43:05.450456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:05.450312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:05.468814 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:05.468757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" podStartSLOduration=2.8504086429999997 podStartE2EDuration="3.468737237s" totalImagesPullingTime="618.328594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:05.467832079 +0000 UTC m=+2372.766309834" watchObservedRunningTime="2026-07-29 21:43:05.468737237 +0000 UTC m=+2372.767214991"
Jul 29 21:43:11.462190 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:11.462161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:43:15.452646 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:15.452615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:43:15.453103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:15.452937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:43:31.841104 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.841060    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:43:31.845640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.845617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:31.848151 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.848128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8cd3f9-serving-cert\""
Jul 29 21:43:31.853000 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.852709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:43:31.952508 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.952467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:31.952508 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.952511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:31.952744 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:31.952554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkm97\" (UniqueName: \"kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.053288 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.053246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.053476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.053298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkm97\" (UniqueName: \"kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.053476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.053382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.054043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.054024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.056270 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.056249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.071819 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.071780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkm97\" (UniqueName: \"kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97\") pod \"maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.159672 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.159630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:32.313266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.313229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:43:32.314599 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:32.314570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6123db08_122d_40b7_ab3b_c26266decce6.slice/crio-b2c66f10495bd6b5a3abc8b5f9897bfc5ea0e92273f0e16a128ab19242c1298c WatchSource:0}: Error finding container b2c66f10495bd6b5a3abc8b5f9897bfc5ea0e92273f0e16a128ab19242c1298c: Status 404 returned error can't find the container with id b2c66f10495bd6b5a3abc8b5f9897bfc5ea0e92273f0e16a128ab19242c1298c
Jul 29 21:43:32.603752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.603719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" event={"ID":"6123db08-122d-40b7-ab3b-c26266decce6","Type":"ContainerStarted","Data":"b2c66f10495bd6b5a3abc8b5f9897bfc5ea0e92273f0e16a128ab19242c1298c"}
Jul 29 21:43:32.823810 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.823767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:43:32.827405 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.827386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:32.830230 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.830199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8cd3f9-dockercfg-nx5lb\""
Jul 29 21:43:32.830230 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.830218    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8cd3f9\""
Jul 29 21:43:32.838717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.838673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:43:32.867347 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.867262    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:43:32.871044 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.871020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:32.880329 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.880299    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:43:32.963950 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.963902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw2r\" (UniqueName: \"kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:32.964140 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.963991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:32.964140 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.964039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:32.964255 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:32.964154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skr7x\" (UniqueName: \"kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.065495 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.065460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skr7x\" (UniqueName: \"kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.065690 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.065553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw2r\" (UniqueName: \"kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.065808 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.065746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.065882 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.065817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.068898 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.068871    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8cd3f9\""
Jul 29 21:43:33.075172 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.075144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skr7x\" (UniqueName: \"kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.075355 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.075316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw2r\" (UniqueName: \"kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.077625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.077602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.077625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.077615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.142090 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.141995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8cd3f9-dockercfg-nx5lb\""
Jul 29 21:43:33.150170 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.150138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.184884 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.184847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.327708 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.326659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:43:33.329689 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.329506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:43:33.332596 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:33.332175    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6343df2c_6fd1_4869_8813_ecfe3620b70b.slice/crio-56002cc4f1bb7e93f3dfeace975e0e0036826e6ab85f85d6c07321c73023152c WatchSource:0}: Error finding container 56002cc4f1bb7e93f3dfeace975e0e0036826e6ab85f85d6c07321c73023152c: Status 404 returned error can't find the container with id 56002cc4f1bb7e93f3dfeace975e0e0036826e6ab85f85d6c07321c73023152c
Jul 29 21:43:33.337535 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.337512    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:43:33.363328 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.363279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:43:33.366099 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:43:33.366073    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba0277e_79cb_48b3_9b52_35ba17936b1b.slice/crio-732a5fc693ba380617150aa8917ba6f3f55ae493d57cd332e274ea28758b1a8f WatchSource:0}: Error finding container 732a5fc693ba380617150aa8917ba6f3f55ae493d57cd332e274ea28758b1a8f: Status 404 returned error can't find the container with id 732a5fc693ba380617150aa8917ba6f3f55ae493d57cd332e274ea28758b1a8f
Jul 29 21:43:33.611616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.611581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" event={"ID":"6123db08-122d-40b7-ab3b-c26266decce6","Type":"ContainerStarted","Data":"cf680514a24058221c0c96a07d633461478aeaa46d7963418d811637ea029e74"}
Jul 29 21:43:33.611892 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.611740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:33.613465 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.613430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" event={"ID":"7ba0277e-79cb-48b3-9b52-35ba17936b1b","Type":"ContainerStarted","Data":"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"}
Jul 29 21:43:33.613614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.613487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" event={"ID":"7ba0277e-79cb-48b3-9b52-35ba17936b1b","Type":"ContainerStarted","Data":"732a5fc693ba380617150aa8917ba6f3f55ae493d57cd332e274ea28758b1a8f"}
Jul 29 21:43:33.613614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.613516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:43:33.615080 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.615052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" event={"ID":"6343df2c-6fd1-4869-8813-ecfe3620b70b","Type":"ContainerStarted","Data":"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"}
Jul 29 21:43:33.615232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.615099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" event={"ID":"6343df2c-6fd1-4869-8813-ecfe3620b70b","Type":"ContainerStarted","Data":"56002cc4f1bb7e93f3dfeace975e0e0036826e6ab85f85d6c07321c73023152c"}
Jul 29 21:43:33.615232 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.615134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:33.630750 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.630682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" podStartSLOduration=2.213147272 podStartE2EDuration="2.630661512s" totalImagesPullingTime="417.51424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:33.62888727 +0000 UTC m=+2400.927365025" watchObservedRunningTime="2026-07-29 21:43:33.630661512 +0000 UTC m=+2400.929139266"
Jul 29 21:43:33.646208 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.646103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" podStartSLOduration=1.646085174 podStartE2EDuration="1.646085174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:33.644930357 +0000 UTC m=+2400.943408115" watchObservedRunningTime="2026-07-29 21:43:33.646085174 +0000 UTC m=+2400.944562929"
Jul 29 21:43:33.662016 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:33.661957    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" podStartSLOduration=1.661939556 podStartE2EDuration="1.661939556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:33.659758278 +0000 UTC m=+2400.958236052" watchObservedRunningTime="2026-07-29 21:43:33.661939556 +0000 UTC m=+2400.960417311"
Jul 29 21:43:39.634066 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:39.634029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:43:44.629527 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:44.629493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:43:44.629976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:43:44.629574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:45:37.667634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.667540    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:45:37.668189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.667840    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" podUID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" containerName="maas-api" containerID="cri-o://f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4" gracePeriod=30
Jul 29 21:45:37.719308 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.719266    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:45:37.719723 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.719524    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" podUID="f2716e28-b2f2-4885-ad5b-8770aa60b834" containerName="payload-processing" containerID="cri-o://5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27" gracePeriod=30
Jul 29 21:45:37.731476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.731437    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:45:37.731748 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:37.731724    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" podUID="646a607f-6df2-4a34-a367-1147adb11d52" containerName="payload-pre-processing" containerID="cri-o://57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13" gracePeriod=30
Jul 29 21:45:38.013422 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.013393    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:45:38.051058 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.051027    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:45:38.054628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.054597    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:45:38.142868 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.142834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume\") pod \"646a607f-6df2-4a34-a367-1147adb11d52\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") "
Jul 29 21:45:38.142868 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.142870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdtnz\" (UniqueName: \"kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz\") pod \"646a607f-6df2-4a34-a367-1147adb11d52\" (UID: \"646a607f-6df2-4a34-a367-1147adb11d52\") "
Jul 29 21:45:38.143114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.142904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca\") pod \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") "
Jul 29 21:45:38.143114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.142937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pxnb\" (UniqueName: \"kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb\") pod \"f2716e28-b2f2-4885-ad5b-8770aa60b834\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") "
Jul 29 21:45:38.143114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143017    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume\") pod \"f2716e28-b2f2-4885-ad5b-8770aa60b834\" (UID: \"f2716e28-b2f2-4885-ad5b-8770aa60b834\") "
Jul 29 21:45:38.143114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") pod \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") "
Jul 29 21:45:38.143114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlf97\" (UniqueName: \"kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97\") pod \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\" (UID: \"6c8f491f-89b1-4376-8701-a6e54f0aa1be\") "
Jul 29 21:45:38.143438 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143400    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c8f491f-89b1-4376-8701-a6e54f0aa1be" (UID: "6c8f491f-89b1-4376-8701-a6e54f0aa1be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:38.143516 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "646a607f-6df2-4a34-a367-1147adb11d52" (UID: "646a607f-6df2-4a34-a367-1147adb11d52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:38.143516 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.143479    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2716e28-b2f2-4885-ad5b-8770aa60b834" (UID: "f2716e28-b2f2-4885-ad5b-8770aa60b834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:38.145544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.145508    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c8f491f-89b1-4376-8701-a6e54f0aa1be" (UID: "6c8f491f-89b1-4376-8701-a6e54f0aa1be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:45:38.145675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.145578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97" (OuterVolumeSpecName: "kube-api-access-qlf97") pod "6c8f491f-89b1-4376-8701-a6e54f0aa1be" (UID: "6c8f491f-89b1-4376-8701-a6e54f0aa1be"). InnerVolumeSpecName "kube-api-access-qlf97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:38.145675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.145613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz" (OuterVolumeSpecName: "kube-api-access-cdtnz") pod "646a607f-6df2-4a34-a367-1147adb11d52" (UID: "646a607f-6df2-4a34-a367-1147adb11d52"). InnerVolumeSpecName "kube-api-access-cdtnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:38.145675 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.145629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb" (OuterVolumeSpecName: "kube-api-access-7pxnb") pod "f2716e28-b2f2-4885-ad5b-8770aa60b834" (UID: "f2716e28-b2f2-4885-ad5b-8770aa60b834"). InnerVolumeSpecName "kube-api-access-7pxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244880    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2716e28-b2f2-4885-ad5b-8770aa60b834-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244912    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8f491f-89b1-4376-8701-a6e54f0aa1be-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244921    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlf97\" (UniqueName: \"kubernetes.io/projected/6c8f491f-89b1-4376-8701-a6e54f0aa1be-kube-api-access-qlf97\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244932    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646a607f-6df2-4a34-a367-1147adb11d52-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244940    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdtnz\" (UniqueName: \"kubernetes.io/projected/646a607f-6df2-4a34-a367-1147adb11d52-kube-api-access-cdtnz\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244948    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8f491f-89b1-4376-8701-a6e54f0aa1be-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.244976 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.244956    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pxnb\" (UniqueName: \"kubernetes.io/projected/f2716e28-b2f2-4885-ad5b-8770aa60b834-kube-api-access-7pxnb\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:38.350656 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.350624    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2716e28-b2f2-4885-ad5b-8770aa60b834" containerID="5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27" exitCode=0
Jul 29 21:45:38.350877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.350669    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"
Jul 29 21:45:38.350877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.350733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" event={"ID":"f2716e28-b2f2-4885-ad5b-8770aa60b834","Type":"ContainerDied","Data":"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"}
Jul 29 21:45:38.350877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.350792    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh" event={"ID":"f2716e28-b2f2-4885-ad5b-8770aa60b834","Type":"ContainerDied","Data":"af57161e3b9a72195057315328ac956152d1d0561c715706ad54b3ed06028dd9"}
Jul 29 21:45:38.350877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.350823    2581 scope.go:117] "RemoveContainer" containerID="5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"
Jul 29 21:45:38.352101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.352076    2581 generic.go:356] "Generic (PLEG): container finished" podID="646a607f-6df2-4a34-a367-1147adb11d52" containerID="57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13" exitCode=0
Jul 29 21:45:38.352183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.352118    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"
Jul 29 21:45:38.352183 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.352164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" event={"ID":"646a607f-6df2-4a34-a367-1147adb11d52","Type":"ContainerDied","Data":"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"}
Jul 29 21:45:38.352264 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.352221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm" event={"ID":"646a607f-6df2-4a34-a367-1147adb11d52","Type":"ContainerDied","Data":"bff7c2816240112ca9707279da3b1cd80f9181639cf1f049d2b30b871bd8063b"}
Jul 29 21:45:38.353525 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.353502    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" containerID="f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4" exitCode=0
Jul 29 21:45:38.353633 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.353549    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"
Jul 29 21:45:38.353633 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.353584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" event={"ID":"6c8f491f-89b1-4376-8701-a6e54f0aa1be","Type":"ContainerDied","Data":"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"}
Jul 29 21:45:38.353752 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.353632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2" event={"ID":"6c8f491f-89b1-4376-8701-a6e54f0aa1be","Type":"ContainerDied","Data":"2b0143134675c797717111fcea0faee344d2c560c0c8ee047d6678ae85f72e6a"}
Jul 29 21:45:38.361269 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.361244    2581 scope.go:117] "RemoveContainer" containerID="5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"
Jul 29 21:45:38.361642 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:45:38.361617    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27\": container with ID starting with 5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27 not found: ID does not exist" containerID="5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"
Jul 29 21:45:38.361753 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.361649    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27"} err="failed to get container status \"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27\": rpc error: code = NotFound desc = could not find container \"5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27\": container with ID starting with 5fbe4f35705713d14cf6f5fffdf93f8132e0fe9db9117c16865d9c600635fb27 not found: ID does not exist"
Jul 29 21:45:38.361753 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.361667    2581 scope.go:117] "RemoveContainer" containerID="57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"
Jul 29 21:45:38.373430 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.373409    2581 scope.go:117] "RemoveContainer" containerID="57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"
Jul 29 21:45:38.373765 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:45:38.373743    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13\": container with ID starting with 57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13 not found: ID does not exist" containerID="57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"
Jul 29 21:45:38.373837 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.373773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13"} err="failed to get container status \"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13\": rpc error: code = NotFound desc = could not find container \"57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13\": container with ID starting with 57103475abc37f6bd40c872f3b0ccb13287c97b01e58a797012ccade97cd1b13 not found: ID does not exist"
Jul 29 21:45:38.373837 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.373791    2581 scope.go:117] "RemoveContainer" containerID="f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"
Jul 29 21:45:38.383114 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.383087    2581 scope.go:117] "RemoveContainer" containerID="f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"
Jul 29 21:45:38.383457 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:45:38.383437    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4\": container with ID starting with f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4 not found: ID does not exist" containerID="f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"
Jul 29 21:45:38.383524 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.383464    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4"} err="failed to get container status \"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4\": rpc error: code = NotFound desc = could not find container \"f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4\": container with ID starting with f99317dd2ef9a044557d4fe94f24bb92de5920e78845f36f229b75bec406fee4 not found: ID does not exist"
Jul 29 21:45:38.384245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.384223    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:45:38.388453 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.388423    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c0f1-5bc6cc5c47-h99z2"]
Jul 29 21:45:38.399519 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.399480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:45:38.404056 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.404024    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c0f1-65665d54d4-c7ddm"]
Jul 29 21:45:38.415198 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.415147    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:45:38.420022 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:38.419990    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c0f1-669bf5bd8-78fxh"]
Jul 29 21:45:39.188025 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:39.187991    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646a607f-6df2-4a34-a367-1147adb11d52" path="/var/lib/kubelet/pods/646a607f-6df2-4a34-a367-1147adb11d52/volumes"
Jul 29 21:45:39.188421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:39.188403    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" path="/var/lib/kubelet/pods/6c8f491f-89b1-4376-8701-a6e54f0aa1be/volumes"
Jul 29 21:45:39.188779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:39.188764    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2716e28-b2f2-4885-ad5b-8770aa60b834" path="/var/lib/kubelet/pods/f2716e28-b2f2-4885-ad5b-8770aa60b834/volumes"
Jul 29 21:45:55.299332 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.299291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:45:55.299971 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.299545    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" podUID="6123db08-122d-40b7-ab3b-c26266decce6" containerName="maas-api" containerID="cri-o://cf680514a24058221c0c96a07d633461478aeaa46d7963418d811637ea029e74" gracePeriod=30
Jul 29 21:45:55.352923 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.352870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:45:55.353153 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.353120    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" podUID="6343df2c-6fd1-4869-8813-ecfe3620b70b" containerName="payload-processing" containerID="cri-o://f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde" gracePeriod=30
Jul 29 21:45:55.356135 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.356102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:45:55.356392 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.356363    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" podUID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" containerName="payload-pre-processing" containerID="cri-o://fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94" gracePeriod=30
Jul 29 21:45:55.462680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.462522    2581 generic.go:356] "Generic (PLEG): container finished" podID="6123db08-122d-40b7-ab3b-c26266decce6" containerID="cf680514a24058221c0c96a07d633461478aeaa46d7963418d811637ea029e74" exitCode=0
Jul 29 21:45:55.462680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.462592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" event={"ID":"6123db08-122d-40b7-ab3b-c26266decce6","Type":"ContainerDied","Data":"cf680514a24058221c0c96a07d633461478aeaa46d7963418d811637ea029e74"}
Jul 29 21:45:55.613567 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.613530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:45:55.667684 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.667658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:45:55.672337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.672310    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:45:55.708544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume\") pod \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") "
Jul 29 21:45:55.708544 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skr7x\" (UniqueName: \"kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x\") pod \"6343df2c-6fd1-4869-8813-ecfe3620b70b\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") "
Jul 29 21:45:55.708840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkm97\" (UniqueName: \"kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97\") pod \"6123db08-122d-40b7-ab3b-c26266decce6\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") "
Jul 29 21:45:55.708840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqw2r\" (UniqueName: \"kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r\") pod \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\" (UID: \"7ba0277e-79cb-48b3-9b52-35ba17936b1b\") "
Jul 29 21:45:55.708840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume\") pod \"6343df2c-6fd1-4869-8813-ecfe3620b70b\" (UID: \"6343df2c-6fd1-4869-8813-ecfe3620b70b\") "
Jul 29 21:45:55.709004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca\") pod \"6123db08-122d-40b7-ab3b-c26266decce6\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") "
Jul 29 21:45:55.709004 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.708892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls\") pod \"6123db08-122d-40b7-ab3b-c26266decce6\" (UID: \"6123db08-122d-40b7-ab3b-c26266decce6\") "
Jul 29 21:45:55.709233 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.709202    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6343df2c-6fd1-4869-8813-ecfe3620b70b" (UID: "6343df2c-6fd1-4869-8813-ecfe3620b70b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:55.709504 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.709482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ba0277e-79cb-48b3-9b52-35ba17936b1b" (UID: "7ba0277e-79cb-48b3-9b52-35ba17936b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:55.709564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.709482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6123db08-122d-40b7-ab3b-c26266decce6" (UID: "6123db08-122d-40b7-ab3b-c26266decce6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:55.711107 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.711076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97" (OuterVolumeSpecName: "kube-api-access-tkm97") pod "6123db08-122d-40b7-ab3b-c26266decce6" (UID: "6123db08-122d-40b7-ab3b-c26266decce6"). InnerVolumeSpecName "kube-api-access-tkm97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:55.711107 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.711092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x" (OuterVolumeSpecName: "kube-api-access-skr7x") pod "6343df2c-6fd1-4869-8813-ecfe3620b70b" (UID: "6343df2c-6fd1-4869-8813-ecfe3620b70b"). InnerVolumeSpecName "kube-api-access-skr7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:55.711456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.711438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6123db08-122d-40b7-ab3b-c26266decce6" (UID: "6123db08-122d-40b7-ab3b-c26266decce6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:45:55.711618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.711597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r" (OuterVolumeSpecName: "kube-api-access-cqw2r") pod "7ba0277e-79cb-48b3-9b52-35ba17936b1b" (UID: "7ba0277e-79cb-48b3-9b52-35ba17936b1b"). InnerVolumeSpecName "kube-api-access-cqw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:55.810412 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810377    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkm97\" (UniqueName: \"kubernetes.io/projected/6123db08-122d-40b7-ab3b-c26266decce6-kube-api-access-tkm97\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810412 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw2r\" (UniqueName: \"kubernetes.io/projected/7ba0277e-79cb-48b3-9b52-35ba17936b1b-kube-api-access-cqw2r\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810425    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343df2c-6fd1-4869-8813-ecfe3620b70b-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810434    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6123db08-122d-40b7-ab3b-c26266decce6-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810444    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6123db08-122d-40b7-ab3b-c26266decce6-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810453    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0277e-79cb-48b3-9b52-35ba17936b1b-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.810634 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:55.810462    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skr7x\" (UniqueName: \"kubernetes.io/projected/6343df2c-6fd1-4869-8813-ecfe3620b70b-kube-api-access-skr7x\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:45:56.470059 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.470024    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"
Jul 29 21:45:56.470533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.470023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw" event={"ID":"6123db08-122d-40b7-ab3b-c26266decce6","Type":"ContainerDied","Data":"b2c66f10495bd6b5a3abc8b5f9897bfc5ea0e92273f0e16a128ab19242c1298c"}
Jul 29 21:45:56.470533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.470180    2581 scope.go:117] "RemoveContainer" containerID="cf680514a24058221c0c96a07d633461478aeaa46d7963418d811637ea029e74"
Jul 29 21:45:56.471604 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.471536    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" containerID="fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94" exitCode=0
Jul 29 21:45:56.471604 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.471591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"
Jul 29 21:45:56.471759 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.471586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" event={"ID":"7ba0277e-79cb-48b3-9b52-35ba17936b1b","Type":"ContainerDied","Data":"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"}
Jul 29 21:45:56.471834 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.471773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf" event={"ID":"7ba0277e-79cb-48b3-9b52-35ba17936b1b","Type":"ContainerDied","Data":"732a5fc693ba380617150aa8917ba6f3f55ae493d57cd332e274ea28758b1a8f"}
Jul 29 21:45:56.473486 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.473319    2581 generic.go:356] "Generic (PLEG): container finished" podID="6343df2c-6fd1-4869-8813-ecfe3620b70b" containerID="f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde" exitCode=0
Jul 29 21:45:56.473486 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.473376    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"
Jul 29 21:45:56.473486 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.473384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" event={"ID":"6343df2c-6fd1-4869-8813-ecfe3620b70b","Type":"ContainerDied","Data":"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"}
Jul 29 21:45:56.473486 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.473446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5" event={"ID":"6343df2c-6fd1-4869-8813-ecfe3620b70b","Type":"ContainerDied","Data":"56002cc4f1bb7e93f3dfeace975e0e0036826e6ab85f85d6c07321c73023152c"}
Jul 29 21:45:56.484057 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.483442    2581 scope.go:117] "RemoveContainer" containerID="fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"
Jul 29 21:45:56.494332 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.494310    2581 scope.go:117] "RemoveContainer" containerID="fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"
Jul 29 21:45:56.494634 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:45:56.494612    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94\": container with ID starting with fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94 not found: ID does not exist" containerID="fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"
Jul 29 21:45:56.494733 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.494649    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94"} err="failed to get container status \"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94\": rpc error: code = NotFound desc = could not find container \"fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94\": container with ID starting with fdffe3fd8ad08b6f102c1d6f1245213768aa32c7501eece79c9d86ece54efc94 not found: ID does not exist"
Jul 29 21:45:56.494733 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.494677    2581 scope.go:117] "RemoveContainer" containerID="f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"
Jul 29 21:45:56.501202 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.501168    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:45:56.503468 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.503444    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8cd3f9-d6f559996-sk8mf"]
Jul 29 21:45:56.504564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.504545    2581 scope.go:117] "RemoveContainer" containerID="f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"
Jul 29 21:45:56.504906 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:45:56.504887    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde\": container with ID starting with f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde not found: ID does not exist" containerID="f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"
Jul 29 21:45:56.504970 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.504914    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde"} err="failed to get container status \"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde\": rpc error: code = NotFound desc = could not find container \"f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde\": container with ID starting with f9d8f9a4eab7ac7d8f81e671d2f9f5d40483da22eea0d201cdd32515b53a7fde not found: ID does not exist"
Jul 29 21:45:56.512112 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.512081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:45:56.516080 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.516051    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cd3f9-f8bbf4994-bg5jw"]
Jul 29 21:45:56.526220 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.526185    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:45:56.530444 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:56.530416    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8cd3f9-d6b49bf7d-7brd5"]
Jul 29 21:45:57.187174 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:57.187139    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6123db08-122d-40b7-ab3b-c26266decce6" path="/var/lib/kubelet/pods/6123db08-122d-40b7-ab3b-c26266decce6/volumes"
Jul 29 21:45:57.187533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:57.187518    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6343df2c-6fd1-4869-8813-ecfe3620b70b" path="/var/lib/kubelet/pods/6343df2c-6fd1-4869-8813-ecfe3620b70b/volumes"
Jul 29 21:45:57.187863 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:45:57.187850    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" path="/var/lib/kubelet/pods/7ba0277e-79cb-48b3-9b52-35ba17936b1b/volumes"
Jul 29 21:46:07.839129 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839088    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839460    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646a607f-6df2-4a34-a367-1147adb11d52" containerName="payload-pre-processing"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839472    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="646a607f-6df2-4a34-a367-1147adb11d52" containerName="payload-pre-processing"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6123db08-122d-40b7-ab3b-c26266decce6" containerName="maas-api"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6123db08-122d-40b7-ab3b-c26266decce6" containerName="maas-api"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839500    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" containerName="maas-api"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" containerName="maas-api"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839514    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" containerName="payload-pre-processing"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839520    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" containerName="payload-pre-processing"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839531    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2716e28-b2f2-4885-ad5b-8770aa60b834" containerName="payload-processing"
Jul 29 21:46:07.839533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839536    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2716e28-b2f2-4885-ad5b-8770aa60b834" containerName="payload-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6343df2c-6fd1-4869-8813-ecfe3620b70b" containerName="payload-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839548    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6343df2c-6fd1-4869-8813-ecfe3620b70b" containerName="payload-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839598    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6123db08-122d-40b7-ab3b-c26266decce6" containerName="maas-api"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839606    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8f491f-89b1-4376-8701-a6e54f0aa1be" containerName="maas-api"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839613    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba0277e-79cb-48b3-9b52-35ba17936b1b" containerName="payload-pre-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839621    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2716e28-b2f2-4885-ad5b-8770aa60b834" containerName="payload-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839628    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="646a607f-6df2-4a34-a367-1147adb11d52" containerName="payload-pre-processing"
Jul 29 21:46:07.839924 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.839634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6343df2c-6fd1-4869-8813-ecfe3620b70b" containerName="payload-processing"
Jul 29 21:46:07.844591 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.844567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:07.847238 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.847216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6a8fa4-serving-cert\""
Jul 29 21:46:07.856853 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.856820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:46:07.918115 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.918079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:07.918321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.918139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:07.918321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:07.918269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4fc\" (UniqueName: \"kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.019054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.019011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4fc\" (UniqueName: \"kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.019054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.019058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.019297 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.019100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.019297 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:46:08.019226    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-serving-cert: secret "maas-api-e2e-api-a-6a8fa4-serving-cert" not found
Jul 29 21:46:08.019375 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:46:08.019308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls podName:b8f06eaf-e239-4a57-85b3-7c527fd6bb34 nodeName:}" failed. No retries permitted until 2026-07-29 21:46:08.519285077 +0000 UTC m=+2555.817762816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls") pod "maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" (UID: "b8f06eaf-e239-4a57-85b3-7c527fd6bb34") : secret "maas-api-e2e-api-a-6a8fa4-serving-cert" not found
Jul 29 21:46:08.019779 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.019763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.028029 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.028002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4fc\" (UniqueName: \"kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.516585 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.516547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:46:08.519905 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.519879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.523520 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.523492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.524709 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.524676    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a8fa4\""
Jul 29 21:46:08.524836 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.524691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a8fa4-dockercfg-tpkzp\""
Jul 29 21:46:08.526740 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.526688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") pod \"maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.552968 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.552910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:46:08.553156 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.553040    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:46:08.559168 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.559136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.563349 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.563273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:46:08.625002 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.624963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdm4q\" (UniqueName: \"kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.625178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.625023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.625178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.625085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.625178 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.625136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnht\" (UniqueName: \"kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.725821 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.725789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.726012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.725853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnht\" (UniqueName: \"kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.726012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.725942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdm4q\" (UniqueName: \"kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.726012 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.725989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.726820 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.726796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.726867 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.726796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.734918 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.734893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdm4q\" (UniqueName: \"kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q\") pod \"payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.735052 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.734951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnht\" (UniqueName: \"kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht\") pod \"payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.758461 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.758426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:08.840618 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.840585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:08.872832 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.872579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:08.925028 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.924990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:46:08.929271 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:08.929221    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f06eaf_e239_4a57_85b3_7c527fd6bb34.slice/crio-acec85125986c925445aedf711867fed0cf8d63d964c2b644f069c5a311337e5 WatchSource:0}: Error finding container acec85125986c925445aedf711867fed0cf8d63d964c2b644f069c5a311337e5: Status 404 returned error can't find the container with id acec85125986c925445aedf711867fed0cf8d63d964c2b644f069c5a311337e5
Jul 29 21:46:08.930928 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:08.930908    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:46:09.017564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.017467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:46:09.019985 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:09.019948    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577138df_7ed9_4c10_b7f7_59ce918ecf33.slice/crio-6bd393e7b9d6df2652103bcdb259455b7d36925fe07017806b7698d79f3572b8 WatchSource:0}: Error finding container 6bd393e7b9d6df2652103bcdb259455b7d36925fe07017806b7698d79f3572b8: Status 404 returned error can't find the container with id 6bd393e7b9d6df2652103bcdb259455b7d36925fe07017806b7698d79f3572b8
Jul 29 21:46:09.052237 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.052213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:46:09.063044 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:09.063004    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode682d56f_92aa_4bb3_9350_27fe2afd64a8.slice/crio-52f85c11932e311f0eaaca1bf16a6778472abeccd123f3c94b0efa35ebae9115 WatchSource:0}: Error finding container 52f85c11932e311f0eaaca1bf16a6778472abeccd123f3c94b0efa35ebae9115: Status 404 returned error can't find the container with id 52f85c11932e311f0eaaca1bf16a6778472abeccd123f3c94b0efa35ebae9115
Jul 29 21:46:09.549088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.549051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" event={"ID":"577138df-7ed9-4c10-b7f7-59ce918ecf33","Type":"ContainerStarted","Data":"d390b324c424947703a3301ed8aeead5dffb4937e151ee8d0a79b233822dd3a4"}
Jul 29 21:46:09.549245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.549108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:09.549245 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.549126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" event={"ID":"577138df-7ed9-4c10-b7f7-59ce918ecf33","Type":"ContainerStarted","Data":"6bd393e7b9d6df2652103bcdb259455b7d36925fe07017806b7698d79f3572b8"}
Jul 29 21:46:09.550420 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.550396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" event={"ID":"b8f06eaf-e239-4a57-85b3-7c527fd6bb34","Type":"ContainerStarted","Data":"acec85125986c925445aedf711867fed0cf8d63d964c2b644f069c5a311337e5"}
Jul 29 21:46:09.551783 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.551756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" event={"ID":"e682d56f-92aa-4bb3-9350-27fe2afd64a8","Type":"ContainerStarted","Data":"4a7cfdc20c6d6a092ecb5807add82efafa8781f178a4ede06b495a5487083ebe"}
Jul 29 21:46:09.551905 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.551805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" event={"ID":"e682d56f-92aa-4bb3-9350-27fe2afd64a8","Type":"ContainerStarted","Data":"52f85c11932e311f0eaaca1bf16a6778472abeccd123f3c94b0efa35ebae9115"}
Jul 29 21:46:09.551957 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.551938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:09.565418 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.565357    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" podStartSLOduration=1.56533517 podStartE2EDuration="1.56533517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:09.56399341 +0000 UTC m=+2556.862471169" watchObservedRunningTime="2026-07-29 21:46:09.56533517 +0000 UTC m=+2556.863812928"
Jul 29 21:46:09.581302 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:09.581207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" podStartSLOduration=1.581182276 podStartE2EDuration="1.581182276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:09.58001642 +0000 UTC m=+2556.878494178" watchObservedRunningTime="2026-07-29 21:46:09.581182276 +0000 UTC m=+2556.879660032"
Jul 29 21:46:10.561454 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:10.561413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" event={"ID":"b8f06eaf-e239-4a57-85b3-7c527fd6bb34","Type":"ContainerStarted","Data":"5502cfb1230b66ba6734e04b0f926fcf1f8dfbb800d5a2ab589b9e62d3fa0dbc"}
Jul 29 21:46:10.561939 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:10.561594    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:10.580485 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:10.580435    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" podStartSLOduration=3.014564184 podStartE2EDuration="3.580418181s" totalImagesPullingTime="565.853997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:10.579210711 +0000 UTC m=+2557.877688466" watchObservedRunningTime="2026-07-29 21:46:10.580418181 +0000 UTC m=+2557.878895990"
Jul 29 21:46:16.574207 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:16.574176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:46:20.564375 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:20.564341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:46:20.564764 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:20.564422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:46:36.846089 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.846052    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:46:36.851391 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.851368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.854070 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.854040    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-881a99-serving-cert\""
Jul 29 21:46:36.862184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.862105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:46:36.885261 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.885217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.885466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.885293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.885466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.885322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.986294 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.986258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.986490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.986328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.986490 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.986355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.986490 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:46:36.986407    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-serving-cert: secret "maas-api-e2e-api-b-881a99-serving-cert" not found
Jul 29 21:46:36.986490 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:46:36.986471    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls podName:8a785213-970d-46a2-8f9f-717331a0cb40 nodeName:}" failed. No retries permitted until 2026-07-29 21:46:37.486451546 +0000 UTC m=+2584.784929279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls") pod "maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" (UID: "8a785213-970d-46a2-8f9f-717331a0cb40") : secret "maas-api-e2e-api-b-881a99-serving-cert" not found
Jul 29 21:46:36.987054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.987035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:36.999108 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:36.999078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:37.492716 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.492661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:37.495717 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.495678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") pod \"maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:37.611446 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.611403    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:46:37.615321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.615301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.617935 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.617907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-881a99-dockercfg-tdsd6\""
Jul 29 21:46:37.618057 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.617917    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-881a99\""
Jul 29 21:46:37.632859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.632825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:46:37.635482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.635451    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:46:37.639046 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.639021    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.651913 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.651741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:46:37.694158 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.694122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zgg\" (UniqueName: \"kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.694158 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.694169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwlf\" (UniqueName: \"kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.694399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.694199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.694399 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.694288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.766189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.766093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:37.794968 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.794926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zgg\" (UniqueName: \"kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.795166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.794996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwlf\" (UniqueName: \"kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.795166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.795046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.795300 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.795232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.796103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.796076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.796103 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.796109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.804911 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.804883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwlf\" (UniqueName: \"kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf\") pod \"payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") " pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:37.805080 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.804988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zgg\" (UniqueName: \"kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg\") pod \"payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.923497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.923454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:46:37.926097 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.926075    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:37.926422 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:37.926398    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a785213_970d_46a2_8f9f_717331a0cb40.slice/crio-a8f3c9471e2b90986091f00af17897625017538207afc3e6ff7776b4c423ed84 WatchSource:0}: Error finding container a8f3c9471e2b90986091f00af17897625017538207afc3e6ff7776b4c423ed84: Status 404 returned error can't find the container with id a8f3c9471e2b90986091f00af17897625017538207afc3e6ff7776b4c423ed84
Jul 29 21:46:37.950897 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:37.950864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:38.098474 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.098425    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:46:38.100875 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:38.100830    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b183ef_db8e_4693_a8b8_1b02fc4ff887.slice/crio-b28145303207fb3bb0882b2f988ccd7e70659c4ac4c06c9969d4e2e08f3b16b1 WatchSource:0}: Error finding container b28145303207fb3bb0882b2f988ccd7e70659c4ac4c06c9969d4e2e08f3b16b1: Status 404 returned error can't find the container with id b28145303207fb3bb0882b2f988ccd7e70659c4ac4c06c9969d4e2e08f3b16b1
Jul 29 21:46:38.134658 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.134632    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:46:38.144066 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:46:38.144034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5cf635_a1ef_40ca_b9be_dd3a88efa617.slice/crio-f17360ade8d09f61cae3893405f0c789cc202da6d19d0602ed28076261cf3efa WatchSource:0}: Error finding container f17360ade8d09f61cae3893405f0c789cc202da6d19d0602ed28076261cf3efa: Status 404 returned error can't find the container with id f17360ade8d09f61cae3893405f0c789cc202da6d19d0602ed28076261cf3efa
Jul 29 21:46:38.728605 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.728562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" event={"ID":"8a785213-970d-46a2-8f9f-717331a0cb40","Type":"ContainerStarted","Data":"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"}
Jul 29 21:46:38.728895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.728642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:38.728895 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.728666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" event={"ID":"8a785213-970d-46a2-8f9f-717331a0cb40","Type":"ContainerStarted","Data":"a8f3c9471e2b90986091f00af17897625017538207afc3e6ff7776b4c423ed84"}
Jul 29 21:46:38.730321 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.730290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" event={"ID":"95b183ef-db8e-4693-a8b8-1b02fc4ff887","Type":"ContainerStarted","Data":"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"}
Jul 29 21:46:38.730483 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.730347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" event={"ID":"95b183ef-db8e-4693-a8b8-1b02fc4ff887","Type":"ContainerStarted","Data":"b28145303207fb3bb0882b2f988ccd7e70659c4ac4c06c9969d4e2e08f3b16b1"}
Jul 29 21:46:38.730483 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.730375    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:38.731896 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.731867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" event={"ID":"3d5cf635-a1ef-40ca-b9be-dd3a88efa617","Type":"ContainerStarted","Data":"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"}
Jul 29 21:46:38.732027 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.731923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" event={"ID":"3d5cf635-a1ef-40ca-b9be-dd3a88efa617","Type":"ContainerStarted","Data":"f17360ade8d09f61cae3893405f0c789cc202da6d19d0602ed28076261cf3efa"}
Jul 29 21:46:38.732027 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.731951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:46:38.790526 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.790456    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" podStartSLOduration=1.790437893 podStartE2EDuration="1.790437893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:38.789077287 +0000 UTC m=+2586.087555042" watchObservedRunningTime="2026-07-29 21:46:38.790437893 +0000 UTC m=+2586.088915648"
Jul 29 21:46:38.790850 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.790826    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" podStartSLOduration=2.419092005 podStartE2EDuration="2.790818045s" totalImagesPullingTime="371.72604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:38.753109438 +0000 UTC m=+2586.051587192" watchObservedRunningTime="2026-07-29 21:46:38.790818045 +0000 UTC m=+2586.089295801"
Jul 29 21:46:38.815999 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:38.815942    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" podStartSLOduration=1.8159226579999999 podStartE2EDuration="1.815922658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:38.815590805 +0000 UTC m=+2586.114068559" watchObservedRunningTime="2026-07-29 21:46:38.815922658 +0000 UTC m=+2586.114400412"
Jul 29 21:46:44.743673 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:44.743643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:46:49.739713 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:49.739656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:46:49.740139 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:46:49.739770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:47:19.652660 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.652568    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:47:19.653249 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.652877    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" podUID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" containerName="maas-api" containerID="cri-o://5502cfb1230b66ba6734e04b0f926fcf1f8dfbb800d5a2ab589b9e62d3fa0dbc" gracePeriod=30
Jul 29 21:47:19.712909 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.712870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:47:19.713224 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.713155    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" podUID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" containerName="payload-processing" containerID="cri-o://4a7cfdc20c6d6a092ecb5807add82efafa8781f178a4ede06b495a5487083ebe" gracePeriod=30
Jul 29 21:47:19.721680 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.721641    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:47:19.721943 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.721914    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" podUID="577138df-7ed9-4c10-b7f7-59ce918ecf33" containerName="payload-pre-processing" containerID="cri-o://d390b324c424947703a3301ed8aeead5dffb4937e151ee8d0a79b233822dd3a4" gracePeriod=30
Jul 29 21:47:19.981788 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.981744    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" containerID="5502cfb1230b66ba6734e04b0f926fcf1f8dfbb800d5a2ab589b9e62d3fa0dbc" exitCode=0
Jul 29 21:47:19.981974 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.981822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" event={"ID":"b8f06eaf-e239-4a57-85b3-7c527fd6bb34","Type":"ContainerDied","Data":"5502cfb1230b66ba6734e04b0f926fcf1f8dfbb800d5a2ab589b9e62d3fa0dbc"}
Jul 29 21:47:19.985128 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.984978    2581 generic.go:356] "Generic (PLEG): container finished" podID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" containerID="4a7cfdc20c6d6a092ecb5807add82efafa8781f178a4ede06b495a5487083ebe" exitCode=0
Jul 29 21:47:19.985128 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.985028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" event={"ID":"e682d56f-92aa-4bb3-9350-27fe2afd64a8","Type":"ContainerDied","Data":"4a7cfdc20c6d6a092ecb5807add82efafa8781f178a4ede06b495a5487083ebe"}
Jul 29 21:47:19.987378 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.987344    2581 generic.go:356] "Generic (PLEG): container finished" podID="577138df-7ed9-4c10-b7f7-59ce918ecf33" containerID="d390b324c424947703a3301ed8aeead5dffb4937e151ee8d0a79b233822dd3a4" exitCode=0
Jul 29 21:47:19.987538 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:19.987397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" event={"ID":"577138df-7ed9-4c10-b7f7-59ce918ecf33","Type":"ContainerDied","Data":"d390b324c424947703a3301ed8aeead5dffb4937e151ee8d0a79b233822dd3a4"}
Jul 29 21:47:20.006088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.006063    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:47:20.031972 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.031945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:47:20.049805 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.049768    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:47:20.096722 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096657    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") pod \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") "
Jul 29 21:47:20.096722 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume\") pod \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") "
Jul 29 21:47:20.096994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcnht\" (UniqueName: \"kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht\") pod \"577138df-7ed9-4c10-b7f7-59ce918ecf33\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") "
Jul 29 21:47:20.096994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdm4q\" (UniqueName: \"kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q\") pod \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\" (UID: \"e682d56f-92aa-4bb3-9350-27fe2afd64a8\") "
Jul 29 21:47:20.096994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume\") pod \"577138df-7ed9-4c10-b7f7-59ce918ecf33\" (UID: \"577138df-7ed9-4c10-b7f7-59ce918ecf33\") "
Jul 29 21:47:20.096994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca\") pod \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") "
Jul 29 21:47:20.096994 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.096931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml4fc\" (UniqueName: \"kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc\") pod \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\" (UID: \"b8f06eaf-e239-4a57-85b3-7c527fd6bb34\") "
Jul 29 21:47:20.097327 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.097296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e682d56f-92aa-4bb3-9350-27fe2afd64a8" (UID: "e682d56f-92aa-4bb3-9350-27fe2afd64a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:20.097789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.097497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8f06eaf-e239-4a57-85b3-7c527fd6bb34" (UID: "b8f06eaf-e239-4a57-85b3-7c527fd6bb34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:20.097789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.097750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "577138df-7ed9-4c10-b7f7-59ce918ecf33" (UID: "577138df-7ed9-4c10-b7f7-59ce918ecf33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:20.099339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.099312    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8f06eaf-e239-4a57-85b3-7c527fd6bb34" (UID: "b8f06eaf-e239-4a57-85b3-7c527fd6bb34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:47:20.099448 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.099397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc" (OuterVolumeSpecName: "kube-api-access-ml4fc") pod "b8f06eaf-e239-4a57-85b3-7c527fd6bb34" (UID: "b8f06eaf-e239-4a57-85b3-7c527fd6bb34"). InnerVolumeSpecName "kube-api-access-ml4fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:20.099626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.099607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q" (OuterVolumeSpecName: "kube-api-access-qdm4q") pod "e682d56f-92aa-4bb3-9350-27fe2afd64a8" (UID: "e682d56f-92aa-4bb3-9350-27fe2afd64a8"). InnerVolumeSpecName "kube-api-access-qdm4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:20.099747 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.099732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht" (OuterVolumeSpecName: "kube-api-access-bcnht") pod "577138df-7ed9-4c10-b7f7-59ce918ecf33" (UID: "577138df-7ed9-4c10-b7f7-59ce918ecf33"). InnerVolumeSpecName "kube-api-access-bcnht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197888    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcnht\" (UniqueName: \"kubernetes.io/projected/577138df-7ed9-4c10-b7f7-59ce918ecf33-kube-api-access-bcnht\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197914    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdm4q\" (UniqueName: \"kubernetes.io/projected/e682d56f-92aa-4bb3-9350-27fe2afd64a8-kube-api-access-qdm4q\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197929    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/577138df-7ed9-4c10-b7f7-59ce918ecf33-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197941    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197950    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4fc\" (UniqueName: \"kubernetes.io/projected/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-kube-api-access-ml4fc\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197960    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f06eaf-e239-4a57-85b3-7c527fd6bb34-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.197982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.197968    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e682d56f-92aa-4bb3-9350-27fe2afd64a8-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:20.994602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.994506    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"
Jul 29 21:47:20.994602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.994547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp" event={"ID":"577138df-7ed9-4c10-b7f7-59ce918ecf33","Type":"ContainerDied","Data":"6bd393e7b9d6df2652103bcdb259455b7d36925fe07017806b7698d79f3572b8"}
Jul 29 21:47:20.995145 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.994610    2581 scope.go:117] "RemoveContainer" containerID="d390b324c424947703a3301ed8aeead5dffb4937e151ee8d0a79b233822dd3a4"
Jul 29 21:47:20.996108 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.996089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"
Jul 29 21:47:20.996193 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.996103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw" event={"ID":"b8f06eaf-e239-4a57-85b3-7c527fd6bb34","Type":"ContainerDied","Data":"acec85125986c925445aedf711867fed0cf8d63d964c2b644f069c5a311337e5"}
Jul 29 21:47:20.997628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.997608    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"
Jul 29 21:47:20.997765 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:20.997605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46" event={"ID":"e682d56f-92aa-4bb3-9350-27fe2afd64a8","Type":"ContainerDied","Data":"52f85c11932e311f0eaaca1bf16a6778472abeccd123f3c94b0efa35ebae9115"}
Jul 29 21:47:21.007827 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.007744    2581 scope.go:117] "RemoveContainer" containerID="5502cfb1230b66ba6734e04b0f926fcf1f8dfbb800d5a2ab589b9e62d3fa0dbc"
Jul 29 21:47:21.020335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.020308    2581 scope.go:117] "RemoveContainer" containerID="4a7cfdc20c6d6a092ecb5807add82efafa8781f178a4ede06b495a5487083ebe"
Jul 29 21:47:21.025023 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.024994    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:47:21.028945 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.028915    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a8fa4-7f665966d4-4g4vp"]
Jul 29 21:47:21.040043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.040012    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:47:21.043531 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.043503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a8fa4-86c6ddb4dc-h6xvw"]
Jul 29 21:47:21.055084 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.055052    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:47:21.059255 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.059223    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a8fa4-7d5dbd9996-88w46"]
Jul 29 21:47:21.188348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.188316    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577138df-7ed9-4c10-b7f7-59ce918ecf33" path="/var/lib/kubelet/pods/577138df-7ed9-4c10-b7f7-59ce918ecf33/volumes"
Jul 29 21:47:21.188748 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.188733    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" path="/var/lib/kubelet/pods/b8f06eaf-e239-4a57-85b3-7c527fd6bb34/volumes"
Jul 29 21:47:21.189054 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:21.189042    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" path="/var/lib/kubelet/pods/e682d56f-92aa-4bb3-9350-27fe2afd64a8/volumes"
Jul 29 21:47:42.282815 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.282777    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:47:42.283299 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.283089    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" podUID="8a785213-970d-46a2-8f9f-717331a0cb40" containerName="maas-api" containerID="cri-o://6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de" gracePeriod=30
Jul 29 21:47:42.348485 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.348434    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:47:42.348780 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.348748    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" podUID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" containerName="payload-processing" containerID="cri-o://736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9" gracePeriod=30
Jul 29 21:47:42.351749 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.351715    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:47:42.351979 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.351949    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" podUID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" containerName="payload-pre-processing" containerID="cri-o://cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0" gracePeriod=30
Jul 29 21:47:42.640888 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.640857    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:47:42.646152 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.646124    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:47:42.673649 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.673619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:47:42.693979 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.693947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg\") pod \"8a785213-970d-46a2-8f9f-717331a0cb40\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") "
Jul 29 21:47:42.694157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") pod \"8a785213-970d-46a2-8f9f-717331a0cb40\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") "
Jul 29 21:47:42.694157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca\") pod \"8a785213-970d-46a2-8f9f-717331a0cb40\" (UID: \"8a785213-970d-46a2-8f9f-717331a0cb40\") "
Jul 29 21:47:42.694157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hwlf\" (UniqueName: \"kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf\") pod \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") "
Jul 29 21:47:42.694157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume\") pod \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\" (UID: \"3d5cf635-a1ef-40ca-b9be-dd3a88efa617\") "
Jul 29 21:47:42.694476 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a785213-970d-46a2-8f9f-717331a0cb40" (UID: "8a785213-970d-46a2-8f9f-717331a0cb40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:42.694560 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.694513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d5cf635-a1ef-40ca-b9be-dd3a88efa617" (UID: "3d5cf635-a1ef-40ca-b9be-dd3a88efa617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:42.696334 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.696306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a785213-970d-46a2-8f9f-717331a0cb40" (UID: "8a785213-970d-46a2-8f9f-717331a0cb40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:47:42.696916 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.696894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf" (OuterVolumeSpecName: "kube-api-access-6hwlf") pod "3d5cf635-a1ef-40ca-b9be-dd3a88efa617" (UID: "3d5cf635-a1ef-40ca-b9be-dd3a88efa617"). InnerVolumeSpecName "kube-api-access-6hwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:42.696987 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.696909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg" (OuterVolumeSpecName: "kube-api-access-2bhxg") pod "8a785213-970d-46a2-8f9f-717331a0cb40" (UID: "8a785213-970d-46a2-8f9f-717331a0cb40"). InnerVolumeSpecName "kube-api-access-2bhxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:42.795246 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795212    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7zgg\" (UniqueName: \"kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg\") pod \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") "
Jul 29 21:47:42.795433 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume\") pod \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\" (UID: \"95b183ef-db8e-4693-a8b8-1b02fc4ff887\") "
Jul 29 21:47:42.795593 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795573    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a785213-970d-46a2-8f9f-717331a0cb40-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.795640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795604    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwlf\" (UniqueName: \"kubernetes.io/projected/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-kube-api-access-6hwlf\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.795640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795619    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5cf635-a1ef-40ca-b9be-dd3a88efa617-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.795640 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/8a785213-970d-46a2-8f9f-717331a0cb40-kube-api-access-2bhxg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.795775 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795650    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a785213-970d-46a2-8f9f-717331a0cb40-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.795775 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.795670    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95b183ef-db8e-4693-a8b8-1b02fc4ff887" (UID: "95b183ef-db8e-4693-a8b8-1b02fc4ff887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:47:42.797566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.797515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg" (OuterVolumeSpecName: "kube-api-access-r7zgg") pod "95b183ef-db8e-4693-a8b8-1b02fc4ff887" (UID: "95b183ef-db8e-4693-a8b8-1b02fc4ff887"). InnerVolumeSpecName "kube-api-access-r7zgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:47:42.896091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.896053    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7zgg\" (UniqueName: \"kubernetes.io/projected/95b183ef-db8e-4693-a8b8-1b02fc4ff887-kube-api-access-r7zgg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:42.896091 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:42.896082    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b183ef-db8e-4693-a8b8-1b02fc4ff887-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:47:43.128571 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.128530    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a785213-970d-46a2-8f9f-717331a0cb40" containerID="6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de" exitCode=0
Jul 29 21:47:43.128789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.128583    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"
Jul 29 21:47:43.128789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.128627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" event={"ID":"8a785213-970d-46a2-8f9f-717331a0cb40","Type":"ContainerDied","Data":"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"}
Jul 29 21:47:43.128789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.128687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s" event={"ID":"8a785213-970d-46a2-8f9f-717331a0cb40","Type":"ContainerDied","Data":"a8f3c9471e2b90986091f00af17897625017538207afc3e6ff7776b4c423ed84"}
Jul 29 21:47:43.128789 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.128732    2581 scope.go:117] "RemoveContainer" containerID="6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"
Jul 29 21:47:43.130050 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.130023    2581 generic.go:356] "Generic (PLEG): container finished" podID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" containerID="cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0" exitCode=0
Jul 29 21:47:43.130205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.130057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" event={"ID":"95b183ef-db8e-4693-a8b8-1b02fc4ff887","Type":"ContainerDied","Data":"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"}
Jul 29 21:47:43.130205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.130110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj" event={"ID":"95b183ef-db8e-4693-a8b8-1b02fc4ff887","Type":"ContainerDied","Data":"b28145303207fb3bb0882b2f988ccd7e70659c4ac4c06c9969d4e2e08f3b16b1"}
Jul 29 21:47:43.130205 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.130068    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"
Jul 29 21:47:43.131496 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.131471    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" containerID="736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9" exitCode=0
Jul 29 21:47:43.131626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.131493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" event={"ID":"3d5cf635-a1ef-40ca-b9be-dd3a88efa617","Type":"ContainerDied","Data":"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"}
Jul 29 21:47:43.131626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.131517    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"
Jul 29 21:47:43.131626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.131532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2" event={"ID":"3d5cf635-a1ef-40ca-b9be-dd3a88efa617","Type":"ContainerDied","Data":"f17360ade8d09f61cae3893405f0c789cc202da6d19d0602ed28076261cf3efa"}
Jul 29 21:47:43.139688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.139668    2581 scope.go:117] "RemoveContainer" containerID="6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"
Jul 29 21:47:43.140070 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:47:43.140041    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de\": container with ID starting with 6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de not found: ID does not exist" containerID="6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"
Jul 29 21:47:43.140160 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.140085    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de"} err="failed to get container status \"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de\": rpc error: code = NotFound desc = could not find container \"6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de\": container with ID starting with 6b133d66a2feb66de53b476a2ca5dc55126739fd7e30eb9ee8574740792720de not found: ID does not exist"
Jul 29 21:47:43.140160 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.140110    2581 scope.go:117] "RemoveContainer" containerID="cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"
Jul 29 21:47:43.152566 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.152537    2581 scope.go:117] "RemoveContainer" containerID="cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"
Jul 29 21:47:43.152927 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:47:43.152905    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0\": container with ID starting with cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0 not found: ID does not exist" containerID="cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"
Jul 29 21:47:43.152981 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.152937    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0"} err="failed to get container status \"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0\": rpc error: code = NotFound desc = could not find container \"cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0\": container with ID starting with cdf39f12850b5efc9719fee1f7d6214e588518404cce19bb7052dc8a8faac2a0 not found: ID does not exist"
Jul 29 21:47:43.152981 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.152955    2581 scope.go:117] "RemoveContainer" containerID="736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"
Jul 29 21:47:43.158885 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.158857    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:47:43.162722 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.162588    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-881a99-6d9645c4c8-5qc5s"]
Jul 29 21:47:43.162797 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.162739    2581 scope.go:117] "RemoveContainer" containerID="736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"
Jul 29 21:47:43.163092 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:47:43.163065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9\": container with ID starting with 736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9 not found: ID does not exist" containerID="736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"
Jul 29 21:47:43.163208 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.163104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9"} err="failed to get container status \"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9\": rpc error: code = NotFound desc = could not find container \"736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9\": container with ID starting with 736ac6cad2907c77f48204d272d50e9ab88c0cdd56c06c85a2e14a0c5596ffc9 not found: ID does not exist"
Jul 29 21:47:43.170378 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.170344    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:47:43.174371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.174343    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-881a99-5d4b5d4d58-tbnq2"]
Jul 29 21:47:43.187483 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.187448    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" path="/var/lib/kubelet/pods/3d5cf635-a1ef-40ca-b9be-dd3a88efa617/volumes"
Jul 29 21:47:43.188325 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.188299    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a785213-970d-46a2-8f9f-717331a0cb40" path="/var/lib/kubelet/pods/8a785213-970d-46a2-8f9f-717331a0cb40/volumes"
Jul 29 21:47:43.188659 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.188643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:47:43.190101 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:43.190084    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-881a99-bc765f948-md6hj"]
Jul 29 21:47:45.187794 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:45.187760    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" path="/var/lib/kubelet/pods/95b183ef-db8e-4693-a8b8-1b02fc4ff887/volumes"
Jul 29 21:47:54.289335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289776    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" containerName="payload-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289823    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" containerName="payload-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289836    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a785213-970d-46a2-8f9f-717331a0cb40" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289844    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a785213-970d-46a2-8f9f-717331a0cb40" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289860    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289868    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289889    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577138df-7ed9-4c10-b7f7-59ce918ecf33" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289899    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="577138df-7ed9-4c10-b7f7-59ce918ecf33" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" containerName="payload-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289918    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" containerName="payload-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.289938    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290039    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="577138df-7ed9-4c10-b7f7-59ce918ecf33" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290053    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a785213-970d-46a2-8f9f-717331a0cb40" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290066    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f06eaf-e239-4a57-85b3-7c527fd6bb34" containerName="maas-api"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290075    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b183ef-db8e-4693-a8b8-1b02fc4ff887" containerName="payload-pre-processing"
Jul 29 21:47:54.290068 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290083    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5cf635-a1ef-40ca-b9be-dd3a88efa617" containerName="payload-processing"
Jul 29 21:47:54.290982 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.290089    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e682d56f-92aa-4bb3-9350-27fe2afd64a8" containerName="payload-processing"
Jul 29 21:47:54.295189 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.295165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.297421 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.297400    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fe32bc-serving-cert\""
Jul 29 21:47:54.305908 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.305881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 21:47:54.404962 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.404927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.404962 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.404969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6j57\" (UniqueName: \"kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.405221 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.405076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.506254 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.506211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.506465 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.506299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.506465 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.506323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6j57\" (UniqueName: \"kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.506465 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:47:54.506407    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-serving-cert: secret "maas-api-e2e-shared-a-fe32bc-serving-cert" not found
Jul 29 21:47:54.506645 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:47:54.506511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls podName:11b01367-ab1f-4cd7-a695-7cf463cd2e30 nodeName:}" failed. No retries permitted until 2026-07-29 21:47:55.006488327 +0000 UTC m=+2662.304966074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls") pod "maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" (UID: "11b01367-ab1f-4cd7-a695-7cf463cd2e30") : secret "maas-api-e2e-shared-a-fe32bc-serving-cert" not found
Jul 29 21:47:54.506981 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.506961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:54.515575 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:54.515546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6j57\" (UniqueName: \"kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:55.010339 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.010293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:55.012992 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.012971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") pod \"maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:55.124625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.124579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 21:47:55.128419 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.128401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.130889 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.130860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-fe32bc\""
Jul 29 21:47:55.140324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.140293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 21:47:55.175335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.175298    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 21:47:55.178832 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.178813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.181565 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.181541    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-fe32bc-dockercfg-nnnbt\""
Jul 29 21:47:55.191115 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.191089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 21:47:55.209176 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.209147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:55.211540 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.211512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdglk\" (UniqueName: \"kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.211613 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.211551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.312855 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.312822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdglk\" (UniqueName: \"kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.313333 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.312860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.313333 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.312918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.313333 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.312946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8fq\" (UniqueName: \"kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.313877 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.313848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.325374 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.325342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdglk\" (UniqueName: \"kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk\") pod \"payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.364266 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.364234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 21:47:55.366160 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:47:55.366135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b01367_ab1f_4cd7_a695_7cf463cd2e30.slice/crio-b454d9528c8a0463e44f6809e090e1ae7abcbf400e2eddf21687fd39e3968102 WatchSource:0}: Error finding container b454d9528c8a0463e44f6809e090e1ae7abcbf400e2eddf21687fd39e3968102: Status 404 returned error can't find the container with id b454d9528c8a0463e44f6809e090e1ae7abcbf400e2eddf21687fd39e3968102
Jul 29 21:47:55.413664 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.413627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.413875 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.413669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8fq\" (UniqueName: \"kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.414402 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.414382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.422564 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.422533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8fq\" (UniqueName: \"kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq\") pod \"payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.441385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.441353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:55.491533 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.491497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:55.602541 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.602443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 21:47:55.662364 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:55.662332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 21:47:55.663804 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:47:55.663770    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2456de57_c341_479d_bb63_3e9265abb921.slice/crio-ca08352be0bc6d59c4408e5c40a30e7de2efb7b3b986256e4842acfbfd4e0ca7 WatchSource:0}: Error finding container ca08352be0bc6d59c4408e5c40a30e7de2efb7b3b986256e4842acfbfd4e0ca7: Status 404 returned error can't find the container with id ca08352be0bc6d59c4408e5c40a30e7de2efb7b3b986256e4842acfbfd4e0ca7
Jul 29 21:47:56.216853 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.216808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" event={"ID":"2456de57-c341-479d-bb63-3e9265abb921","Type":"ContainerStarted","Data":"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"}
Jul 29 21:47:56.217056 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.216887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:47:56.217056 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.216910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" event={"ID":"2456de57-c341-479d-bb63-3e9265abb921","Type":"ContainerStarted","Data":"ca08352be0bc6d59c4408e5c40a30e7de2efb7b3b986256e4842acfbfd4e0ca7"}
Jul 29 21:47:56.218475 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.218446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" event={"ID":"11b01367-ab1f-4cd7-a695-7cf463cd2e30","Type":"ContainerStarted","Data":"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"}
Jul 29 21:47:56.218614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.218503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:47:56.218614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.218530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" event={"ID":"11b01367-ab1f-4cd7-a695-7cf463cd2e30","Type":"ContainerStarted","Data":"b454d9528c8a0463e44f6809e090e1ae7abcbf400e2eddf21687fd39e3968102"}
Jul 29 21:47:56.220241 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.220218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" event={"ID":"8ce36441-292b-49a8-bd6e-61df45d81839","Type":"ContainerStarted","Data":"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"}
Jul 29 21:47:56.220348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.220265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" event={"ID":"8ce36441-292b-49a8-bd6e-61df45d81839","Type":"ContainerStarted","Data":"8e7b680629acb7907cc6436504eae34978696c375aa3971dc780669e840b3a6f"}
Jul 29 21:47:56.220348 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.220286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:47:56.235413 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.235362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" podStartSLOduration=1.235332131 podStartE2EDuration="1.235332131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:56.233659112 +0000 UTC m=+2663.532136869" watchObservedRunningTime="2026-07-29 21:47:56.235332131 +0000 UTC m=+2663.533809888"
Jul 29 21:47:56.251262 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.251186    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" podStartSLOduration=1.251169394 podStartE2EDuration="1.251169394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:56.249669158 +0000 UTC m=+2663.548146914" watchObservedRunningTime="2026-07-29 21:47:56.251169394 +0000 UTC m=+2663.549647172"
Jul 29 21:47:56.268022 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:47:56.267958    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" podStartSLOduration=1.875084194 podStartE2EDuration="2.267935736s" totalImagesPullingTime="392.851542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:56.266430223 +0000 UTC m=+2663.564907983" watchObservedRunningTime="2026-07-29 21:47:56.267935736 +0000 UTC m=+2663.566413493"
Jul 29 21:48:02.231625 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:02.231593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 21:48:07.228250 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:07.228213    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 21:48:07.228645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:07.228406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 21:48:23.632487 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.632396    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 21:48:23.636166 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.636141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.638602 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.638574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-670ddc-serving-cert\""
Jul 29 21:48:23.647466 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.647438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 21:48:23.686133 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.686093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.686335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.686199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.686335 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.686228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xvv\" (UniqueName: \"kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.787088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.787052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.787088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.787094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xvv\" (UniqueName: \"kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.787367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.787135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.787367 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:48:23.787255    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-serving-cert: secret "maas-api-e2e-shared-b-670ddc-serving-cert" not found
Jul 29 21:48:23.787367 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:48:23.787334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls podName:07f39ccf-7cde-4f5b-a14f-8529ad841f61 nodeName:}" failed. No retries permitted until 2026-07-29 21:48:24.287313901 +0000 UTC m=+2691.585791634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls") pod "maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" (UID: "07f39ccf-7cde-4f5b-a14f-8529ad841f61") : secret "maas-api-e2e-shared-b-670ddc-serving-cert" not found
Jul 29 21:48:23.787840 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.787820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:23.798521 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:23.798496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xvv\" (UniqueName: \"kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:24.234263 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.234219    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 21:48:24.240973 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.240936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.243728 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.243679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-670ddc\""
Jul 29 21:48:24.248154 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.248128    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 21:48:24.268337 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.268300    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 21:48:24.271891 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.271871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.274879 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.274855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-670ddc-dockercfg-rm8zc\""
Jul 29 21:48:24.283561 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.283530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 21:48:24.292823 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.292787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.293043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.292843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:24.293043 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.292986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb22h\" (UniqueName: \"kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.295786 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.295762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") pod \"maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:24.394291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.394238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb22h\" (UniqueName: \"kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.394291 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.394286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.394562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.394334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6z26\" (UniqueName: \"kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.394562 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.394359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.395061 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.395038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.402837 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.402801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb22h\" (UniqueName: \"kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h\") pod \"payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.495049 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.494930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6z26\" (UniqueName: \"kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.495251 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.495098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.495843 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.495820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.503955 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.503933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6z26\" (UniqueName: \"kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26\") pod \"payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.550046 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.550007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:24.553157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.553130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:24.583919 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.583883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:24.775401 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.775375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 21:48:24.778030 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:48:24.778001    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9814694_0fe7_4523_a78b_3bf596a6c4aa.slice/crio-b0af7c798f4eb573c2ca85252ea2309da951a7c958f569d60f118dbe4af1cc03 WatchSource:0}: Error finding container b0af7c798f4eb573c2ca85252ea2309da951a7c958f569d60f118dbe4af1cc03: Status 404 returned error can't find the container with id b0af7c798f4eb573c2ca85252ea2309da951a7c958f569d60f118dbe4af1cc03
Jul 29 21:48:24.941937 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.941907    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 21:48:24.944887 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:48:24.944855    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f39ccf_7cde_4f5b_a14f_8529ad841f61.slice/crio-43af537e80c9f2f27d522b086dc9cf25bc0e78e397f0b9b909a1d8eb0d588f9c WatchSource:0}: Error finding container 43af537e80c9f2f27d522b086dc9cf25bc0e78e397f0b9b909a1d8eb0d588f9c: Status 404 returned error can't find the container with id 43af537e80c9f2f27d522b086dc9cf25bc0e78e397f0b9b909a1d8eb0d588f9c
Jul 29 21:48:24.945367 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:24.945257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 21:48:24.946742 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:48:24.946688    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb221664d_4e3a_4841_a563_667856555b38.slice/crio-5a2e3f976bb642a571065feca4365041e9fce5c8c044d76110019b5319a4f1c1 WatchSource:0}: Error finding container 5a2e3f976bb642a571065feca4365041e9fce5c8c044d76110019b5319a4f1c1: Status 404 returned error can't find the container with id 5a2e3f976bb642a571065feca4365041e9fce5c8c044d76110019b5319a4f1c1
Jul 29 21:48:25.391090 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.391049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" event={"ID":"b221664d-4e3a-4841-a563-667856555b38","Type":"ContainerStarted","Data":"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"}
Jul 29 21:48:25.391324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.391113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" event={"ID":"b221664d-4e3a-4841-a563-667856555b38","Type":"ContainerStarted","Data":"5a2e3f976bb642a571065feca4365041e9fce5c8c044d76110019b5319a4f1c1"}
Jul 29 21:48:25.391324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.391257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:25.392825 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.392799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" event={"ID":"b9814694-0fe7-4523-a78b-3bf596a6c4aa","Type":"ContainerStarted","Data":"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"}
Jul 29 21:48:25.392959 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.392843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" event={"ID":"b9814694-0fe7-4523-a78b-3bf596a6c4aa","Type":"ContainerStarted","Data":"b0af7c798f4eb573c2ca85252ea2309da951a7c958f569d60f118dbe4af1cc03"}
Jul 29 21:48:25.392959 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.392880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:48:25.394184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.394157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" event={"ID":"07f39ccf-7cde-4f5b-a14f-8529ad841f61","Type":"ContainerStarted","Data":"43af537e80c9f2f27d522b086dc9cf25bc0e78e397f0b9b909a1d8eb0d588f9c"}
Jul 29 21:48:25.409592 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.409536    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" podStartSLOduration=1.409519009 podStartE2EDuration="1.409519009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:48:25.407979501 +0000 UTC m=+2692.706457255" watchObservedRunningTime="2026-07-29 21:48:25.409519009 +0000 UTC m=+2692.707996797"
Jul 29 21:48:25.427626 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:25.427573    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" podStartSLOduration=1.427555477 podStartE2EDuration="1.427555477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:48:25.425559259 +0000 UTC m=+2692.724037014" watchObservedRunningTime="2026-07-29 21:48:25.427555477 +0000 UTC m=+2692.726033304"
Jul 29 21:48:26.402497 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:26.402460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" event={"ID":"07f39ccf-7cde-4f5b-a14f-8529ad841f61","Type":"ContainerStarted","Data":"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"}
Jul 29 21:48:26.402952 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:26.402670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:26.421705 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:26.421633    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" podStartSLOduration=2.764957657 podStartE2EDuration="3.421614214s" totalImagesPullingTime="656.656557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:48:26.418853807 +0000 UTC m=+2693.717331565" watchObservedRunningTime="2026-07-29 21:48:26.421614214 +0000 UTC m=+2693.720091969"
Jul 29 21:48:32.414340 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:32.414305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 21:48:33.375467 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:33.375436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:48:33.384041 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:33.384013    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:48:36.404426 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:36.404389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 21:48:36.404866 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:48:36.404478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 21:53:33.413320 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:53:33.413291    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:53:33.426357 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:53:33.426330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:56:28.934154 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.934113    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:56:28.937813 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.937796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.941796 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.941772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fe32bc\"/\"default-dockercfg-7gwbz\""
Jul 29 21:56:28.941912 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.941807    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fe32bc\"/\"rate-test-model-fe32bc-kserve-self-signed-certs\""
Jul 29 21:56:28.941912 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.941819    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fe32bc\"/\"kube-root-ca.crt\""
Jul 29 21:56:28.941912 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.941837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fe32bc\"/\"openshift-service-ca.crt\""
Jul 29 21:56:28.949726 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.949678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:56:28.997480 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd27h\" (UniqueName: \"kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.997663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.997663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.997663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.997663 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:28.997842 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:28.997665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099209 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099209 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd27h\" (UniqueName: \"kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099456 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099768 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.099941 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.099861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.101806 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.101777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.101961 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.101943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.108778 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.108730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd27h\" (UniqueName: \"kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h\") pod \"rate-test-model-fe32bc-kserve-7849498787-df4cj\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") " pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.248196 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.248100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:29.405416 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:29.405387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:56:29.408514 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:56:29.408483    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72157f7e_41ad_40e6_af32_f7b166427bbc.slice/crio-f90603c78d0a3bcbb482fd85d93bff90692a9680c720f69a9472c1c77345f6a1 WatchSource:0}: Error finding container f90603c78d0a3bcbb482fd85d93bff90692a9680c720f69a9472c1c77345f6a1: Status 404 returned error can't find the container with id f90603c78d0a3bcbb482fd85d93bff90692a9680c720f69a9472c1c77345f6a1
Jul 29 21:56:30.173434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:30.173395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" event={"ID":"72157f7e-41ad-40e6-af32-f7b166427bbc","Type":"ContainerStarted","Data":"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"}
Jul 29 21:56:30.173434 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:30.173441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" event={"ID":"72157f7e-41ad-40e6-af32-f7b166427bbc","Type":"ContainerStarted","Data":"f90603c78d0a3bcbb482fd85d93bff90692a9680c720f69a9472c1c77345f6a1"}
Jul 29 21:56:30.194804 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:30.194750    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" podStartSLOduration=2.19473343 podStartE2EDuration="2.19473343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:56:30.191813437 +0000 UTC m=+3177.490291193" watchObservedRunningTime="2026-07-29 21:56:30.19473343 +0000 UTC m=+3177.493211185"
Jul 29 21:56:39.248373 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:39.248330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:39.248843 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:39.248649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:39.261492 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:39.261462    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:56:40.240688 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:56:40.240654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:58:15.332766 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:15.332728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:58:30.352961 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.352907    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" podUID="72157f7e-41ad-40e6-af32-f7b166427bbc" containerName="main" containerID="cri-o://a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c" gracePeriod=45
Jul 29 21:58:30.603975 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.603896    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:58:30.764629 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.764859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764647    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.764859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.764859 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764803    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.765028 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd27h\" (UniqueName: \"kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.765028 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home\") pod \"72157f7e-41ad-40e6-af32-f7b166427bbc\" (UID: \"72157f7e-41ad-40e6-af32-f7b166427bbc\") "
Jul 29 21:58:30.765028 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:58:30.765028 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.764947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:58:30.765235 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.765203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home" (OuterVolumeSpecName: "home") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:58:30.765235 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.765211    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-tmp-dir\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.765235 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.765234    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-model-cache\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.767184 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.767151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:58:30.767665 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.767643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h" (OuterVolumeSpecName: "kube-api-access-dd27h") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "kube-api-access-dd27h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:58:30.767778 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.767733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm" (OuterVolumeSpecName: "dshm") pod "72157f7e-41ad-40e6-af32-f7b166427bbc" (UID: "72157f7e-41ad-40e6-af32-f7b166427bbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:58:30.866331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.866239    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72157f7e-41ad-40e6-af32-f7b166427bbc-tls-certs\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.866331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.866268    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-dshm\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.866331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.866278    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd27h\" (UniqueName: \"kubernetes.io/projected/72157f7e-41ad-40e6-af32-f7b166427bbc-kube-api-access-dd27h\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.866331 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.866286    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72157f7e-41ad-40e6-af32-f7b166427bbc-home\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 21:58:30.868628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.868597    2581 generic.go:356] "Generic (PLEG): container finished" podID="72157f7e-41ad-40e6-af32-f7b166427bbc" containerID="a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c" exitCode=0
Jul 29 21:58:30.868777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.868647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"
Jul 29 21:58:30.868777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.868689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" event={"ID":"72157f7e-41ad-40e6-af32-f7b166427bbc","Type":"ContainerDied","Data":"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"}
Jul 29 21:58:30.868777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.868753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj" event={"ID":"72157f7e-41ad-40e6-af32-f7b166427bbc","Type":"ContainerDied","Data":"f90603c78d0a3bcbb482fd85d93bff90692a9680c720f69a9472c1c77345f6a1"}
Jul 29 21:58:30.868777 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.868771    2581 scope.go:117] "RemoveContainer" containerID="a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"
Jul 29 21:58:30.878947 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.878928    2581 scope.go:117] "RemoveContainer" containerID="a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"
Jul 29 21:58:30.879250 ip-10-0-143-168 kubenswrapper[2581]: E0729 21:58:30.879227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c\": container with ID starting with a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c not found: ID does not exist" containerID="a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"
Jul 29 21:58:30.879327 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.879261    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c"} err="failed to get container status \"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c\": rpc error: code = NotFound desc = could not find container \"a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c\": container with ID starting with a2a19292952421fc3071ac09b7614422ffb456c29959e7733e2f1ebc3476cd7c not found: ID does not exist"
Jul 29 21:58:30.891628 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.891597    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:58:30.897465 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:30.897439    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fe32bc/rate-test-model-fe32bc-kserve-7849498787-df4cj"]
Jul 29 21:58:31.188428 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:31.188346    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72157f7e-41ad-40e6-af32-f7b166427bbc" path="/var/lib/kubelet/pods/72157f7e-41ad-40e6-af32-f7b166427bbc/volumes"
Jul 29 21:58:33.449645 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:33.449526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:58:33.465171 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:33.465146    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 21:58:40.763772 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.763733    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 21:58:40.764172 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.764164    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72157f7e-41ad-40e6-af32-f7b166427bbc" containerName="main"
Jul 29 21:58:40.764220 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.764176    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72157f7e-41ad-40e6-af32-f7b166427bbc" containerName="main"
Jul 29 21:58:40.764259 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.764237    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72157f7e-41ad-40e6-af32-f7b166427bbc" containerName="main"
Jul 29 21:58:40.768453 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.768438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.772445 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.772369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a0dedf-dockercfg-kjp47\""
Jul 29 21:58:40.772614 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.772506    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a0dedf\""
Jul 29 21:58:40.785410 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.785380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 21:58:40.802627 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.802590    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 21:58:40.806451 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.806424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.820111 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.820080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 21:58:40.856157 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.856124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglg6\" (UniqueName: \"kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.856404 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.856263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2c7\" (UniqueName: \"kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.856404 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.856294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.856404 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.856325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.957862 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.957807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2c7\" (UniqueName: \"kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.957862 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.957865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.958149 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.957906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.958149 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.957974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglg6\" (UniqueName: \"kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.958896 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.958868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:40.958996 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.958969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.969188 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.969158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglg6\" (UniqueName: \"kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6\") pod \"payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:40.974999 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:40.974965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2c7\" (UniqueName: \"kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7\") pod \"payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:41.080465 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.080364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:41.118163 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.117935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:41.296527 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.296500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 21:58:41.298751 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:58:41.298723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d10d950_852c_45aa_a18a_6e24899e18dd.slice/crio-aa61709405a8afa58e65f309ffaa70866069a90b92c675a464520a7158808ef6 WatchSource:0}: Error finding container aa61709405a8afa58e65f309ffaa70866069a90b92c675a464520a7158808ef6: Status 404 returned error can't find the container with id aa61709405a8afa58e65f309ffaa70866069a90b92c675a464520a7158808ef6
Jul 29 21:58:41.462887 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.462847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 21:58:41.463970 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:58:41.463944    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5510223c_038d_4614_ab98_f8b047a953a8.slice/crio-ccf876caf61c31a5560ca17bc24097f9295acd30aff3d432d4d542a57e6a1646 WatchSource:0}: Error finding container ccf876caf61c31a5560ca17bc24097f9295acd30aff3d432d4d542a57e6a1646: Status 404 returned error can't find the container with id ccf876caf61c31a5560ca17bc24097f9295acd30aff3d432d4d542a57e6a1646
Jul 29 21:58:41.938899 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.938858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" event={"ID":"5510223c-038d-4614-ab98-f8b047a953a8","Type":"ContainerStarted","Data":"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"}
Jul 29 21:58:41.939360 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.938932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" event={"ID":"5510223c-038d-4614-ab98-f8b047a953a8","Type":"ContainerStarted","Data":"ccf876caf61c31a5560ca17bc24097f9295acd30aff3d432d4d542a57e6a1646"}
Jul 29 21:58:41.939360 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.939059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:58:41.940529 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.940504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" event={"ID":"6d10d950-852c-45aa-a18a-6e24899e18dd","Type":"ContainerStarted","Data":"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"}
Jul 29 21:58:41.940668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.940539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" event={"ID":"6d10d950-852c-45aa-a18a-6e24899e18dd","Type":"ContainerStarted","Data":"aa61709405a8afa58e65f309ffaa70866069a90b92c675a464520a7158808ef6"}
Jul 29 21:58:41.940668 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.940569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:41.961733 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.961648    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" podStartSLOduration=1.961625636 podStartE2EDuration="1.961625636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:58:41.956826558 +0000 UTC m=+3309.255304313" watchObservedRunningTime="2026-07-29 21:58:41.961625636 +0000 UTC m=+3309.260103393"
Jul 29 21:58:41.976844 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:41.976788    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" podStartSLOduration=1.976770069 podStartE2EDuration="1.976770069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:58:41.976522377 +0000 UTC m=+3309.275000133" watchObservedRunningTime="2026-07-29 21:58:41.976770069 +0000 UTC m=+3309.275247825"
Jul 29 21:58:52.948194 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:52.948156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 21:58:52.948711 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:58:52.948237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 21:59:09.693283 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.693241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 21:59:09.697210 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.697190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.699995 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.699940    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-50d50c\""
Jul 29 21:59:09.700165 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.700025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-50d50c-dockercfg-wzhtx\""
Jul 29 21:59:09.709390 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.709358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 21:59:09.729088 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.729050    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 21:59:09.733264 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.733243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.739039 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.739011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 21:59:09.834719 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.834661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbvg\" (UniqueName: \"kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.834719 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.834719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.834953 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.834776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn94b\" (UniqueName: \"kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.834953 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.834875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.936332 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.936288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbvg\" (UniqueName: \"kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.936332 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.936336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.936616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.936482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn94b\" (UniqueName: \"kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.936616 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.936541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.937307 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.937284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:09.937371 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.937349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.947344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.947269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbvg\" (UniqueName: \"kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg\") pod \"payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:09.947344 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:09.947269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn94b\" (UniqueName: \"kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b\") pod \"payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:10.009813 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:10.009768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:10.045315 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:10.045271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:10.188296 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:10.188263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 21:59:10.191087 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:59:10.191047    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0d79b3_c8a2_4768_a6ed_2cdbc2a53d89.slice/crio-62cc89464b912bbb47bfa2e74a6e02d8abf41a4ade9ed599bed38a6edde9d9df WatchSource:0}: Error finding container 62cc89464b912bbb47bfa2e74a6e02d8abf41a4ade9ed599bed38a6edde9d9df: Status 404 returned error can't find the container with id 62cc89464b912bbb47bfa2e74a6e02d8abf41a4ade9ed599bed38a6edde9d9df
Jul 29 21:59:10.224385 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:10.224355    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 21:59:10.228900 ip-10-0-143-168 kubenswrapper[2581]: W0729 21:59:10.228865    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7157533d_7192_442c_9d85_5c5f0e20d2af.slice/crio-a6081385af0424a7fb0ee201de788fb1fea9e22499fffb1dc3bb2f31b8fedac8 WatchSource:0}: Error finding container a6081385af0424a7fb0ee201de788fb1fea9e22499fffb1dc3bb2f31b8fedac8: Status 404 returned error can't find the container with id a6081385af0424a7fb0ee201de788fb1fea9e22499fffb1dc3bb2f31b8fedac8
Jul 29 21:59:11.120324 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.120285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" event={"ID":"7157533d-7192-442c-9d85-5c5f0e20d2af","Type":"ContainerStarted","Data":"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"}
Jul 29 21:59:11.120854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.120347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" event={"ID":"7157533d-7192-442c-9d85-5c5f0e20d2af","Type":"ContainerStarted","Data":"a6081385af0424a7fb0ee201de788fb1fea9e22499fffb1dc3bb2f31b8fedac8"}
Jul 29 21:59:11.120854 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.120459    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:11.122104 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.122063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" event={"ID":"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89","Type":"ContainerStarted","Data":"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"}
Jul 29 21:59:11.122231 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.122118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" event={"ID":"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89","Type":"ContainerStarted","Data":"62cc89464b912bbb47bfa2e74a6e02d8abf41a4ade9ed599bed38a6edde9d9df"}
Jul 29 21:59:11.122231 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.122156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:11.140762 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.140716    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" podStartSLOduration=2.140683681 podStartE2EDuration="2.140683681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:59:11.138758546 +0000 UTC m=+3338.437236304" watchObservedRunningTime="2026-07-29 21:59:11.140683681 +0000 UTC m=+3338.439161436"
Jul 29 21:59:11.159590 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:11.159528    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" podStartSLOduration=2.159509153 podStartE2EDuration="2.159509153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:59:11.158210939 +0000 UTC m=+3338.456688695" watchObservedRunningTime="2026-07-29 21:59:11.159509153 +0000 UTC m=+3338.457986908"
Jul 29 21:59:22.129990 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:22.129958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 21:59:22.130386 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:22.130036    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 21:59:39.723482 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:39.723447    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 21:59:39.851272 ip-10-0-143-168 kubenswrapper[2581]: I0729 21:59:39.851244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88_de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89/payload-processing/0.log"
Jul 29 22:00:35.931660 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:35.931625    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr_6d10d950-852c-45aa-a18a-6e24899e18dd/payload-processing/0.log"
Jul 29 22:00:36.073727 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:36.073650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 22:00:36.201215 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:36.201125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88_de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89/payload-processing/0.log"
Jul 29 22:00:41.287638 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.287605    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:00:41.291342 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.291320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.293862 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.293838    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-215280\""
Jul 29 22:00:41.293989 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.293939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-215280-dockercfg-24sfq\""
Jul 29 22:00:41.301499 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.301462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:00:41.325471 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.325425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:00:41.329617 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.329587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.339858 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.339830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:00:41.387578 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.387528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.387827 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.387610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qtq\" (UniqueName: \"kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.488543 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.488509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.488764 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.488553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.488764 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.488649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qtq\" (UniqueName: \"kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.488875 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.488792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whwg\" (UniqueName: \"kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.489356 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.489338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.497430 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.497406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qtq\" (UniqueName: \"kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq\") pod \"payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.589807 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.589686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whwg\" (UniqueName: \"kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.589975 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.589813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.590571 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.590551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.597555 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.597521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whwg\" (UniqueName: \"kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg\") pod \"payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.605504 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.605409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:41.641924 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.641884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:41.822144 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.822114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:00:41.823222 ip-10-0-143-168 kubenswrapper[2581]: W0729 22:00:41.823182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2567df_882c_43bd_9295_0e16bf260f17.slice/crio-158bda9b00357af666557dc85e704cfaeafd282fdd86bd2fc8095a6ad3090297 WatchSource:0}: Error finding container 158bda9b00357af666557dc85e704cfaeafd282fdd86bd2fc8095a6ad3090297: Status 404 returned error can't find the container with id 158bda9b00357af666557dc85e704cfaeafd282fdd86bd2fc8095a6ad3090297
Jul 29 22:00:41.990634 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:41.990605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:00:41.991836 ip-10-0-143-168 kubenswrapper[2581]: W0729 22:00:41.991788    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33f0550_085f_49a5_8acf_18c65e367d9b.slice/crio-564da87575af66132a7b21c572879e1817a6c7f04acbd1caac4cd4765b5a29f5 WatchSource:0}: Error finding container 564da87575af66132a7b21c572879e1817a6c7f04acbd1caac4cd4765b5a29f5: Status 404 returned error can't find the container with id 564da87575af66132a7b21c572879e1817a6c7f04acbd1caac4cd4765b5a29f5
Jul 29 22:00:42.669253 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.669211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" event={"ID":"e33f0550-085f-49a5-8acf-18c65e367d9b","Type":"ContainerStarted","Data":"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"}
Jul 29 22:00:42.669814 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.669326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" event={"ID":"e33f0550-085f-49a5-8acf-18c65e367d9b","Type":"ContainerStarted","Data":"564da87575af66132a7b21c572879e1817a6c7f04acbd1caac4cd4765b5a29f5"}
Jul 29 22:00:42.669814 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.669365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:00:42.671284 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.671251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" event={"ID":"ce2567df-882c-43bd-9295-0e16bf260f17","Type":"ContainerStarted","Data":"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"}
Jul 29 22:00:42.671458 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.671306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" event={"ID":"ce2567df-882c-43bd-9295-0e16bf260f17","Type":"ContainerStarted","Data":"158bda9b00357af666557dc85e704cfaeafd282fdd86bd2fc8095a6ad3090297"}
Jul 29 22:00:42.671458 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.671349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:42.690567 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.690508    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" podStartSLOduration=1.690486288 podStartE2EDuration="1.690486288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:42.68852474 +0000 UTC m=+3429.987002496" watchObservedRunningTime="2026-07-29 22:00:42.690486288 +0000 UTC m=+3429.988964045"
Jul 29 22:00:42.707814 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:42.707758    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" podStartSLOduration=1.707739261 podStartE2EDuration="1.707739261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:42.706971831 +0000 UTC m=+3430.005449587" watchObservedRunningTime="2026-07-29 22:00:42.707739261 +0000 UTC m=+3430.006217018"
Jul 29 22:00:53.680549 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:53.680517    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:00:53.681074 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:00:53.680599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:01:10.354297 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.354249    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:01:10.354877 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.354556    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" podUID="e33f0550-085f-49a5-8acf-18c65e367d9b" containerName="payload-processing" containerID="cri-o://929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d" gracePeriod=30
Jul 29 22:01:10.359439 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.359409    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:01:10.360783 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.360751    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" podUID="ce2567df-882c-43bd-9295-0e16bf260f17" containerName="payload-pre-processing" containerID="cri-o://4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a" gracePeriod=30
Jul 29 22:01:10.676918 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.676891    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:01:10.680224 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.680200    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:01:10.771768 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.771732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume\") pod \"ce2567df-882c-43bd-9295-0e16bf260f17\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") "
Jul 29 22:01:10.771961 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.771793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4whwg\" (UniqueName: \"kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg\") pod \"ce2567df-882c-43bd-9295-0e16bf260f17\" (UID: \"ce2567df-882c-43bd-9295-0e16bf260f17\") "
Jul 29 22:01:10.771961 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.771851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume\") pod \"e33f0550-085f-49a5-8acf-18c65e367d9b\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") "
Jul 29 22:01:10.771961 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.771926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28qtq\" (UniqueName: \"kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq\") pod \"e33f0550-085f-49a5-8acf-18c65e367d9b\" (UID: \"e33f0550-085f-49a5-8acf-18c65e367d9b\") "
Jul 29 22:01:10.772225 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.772197    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce2567df-882c-43bd-9295-0e16bf260f17" (UID: "ce2567df-882c-43bd-9295-0e16bf260f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:01:10.772302 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.772241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e33f0550-085f-49a5-8acf-18c65e367d9b" (UID: "e33f0550-085f-49a5-8acf-18c65e367d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:01:10.774187 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.774160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg" (OuterVolumeSpecName: "kube-api-access-4whwg") pod "ce2567df-882c-43bd-9295-0e16bf260f17" (UID: "ce2567df-882c-43bd-9295-0e16bf260f17"). InnerVolumeSpecName "kube-api-access-4whwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:01:10.774299 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.774169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq" (OuterVolumeSpecName: "kube-api-access-28qtq") pod "e33f0550-085f-49a5-8acf-18c65e367d9b" (UID: "e33f0550-085f-49a5-8acf-18c65e367d9b"). InnerVolumeSpecName "kube-api-access-28qtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:01:10.843185 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.843137    2581 generic.go:356] "Generic (PLEG): container finished" podID="e33f0550-085f-49a5-8acf-18c65e367d9b" containerID="929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d" exitCode=0
Jul 29 22:01:10.843185 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.843191    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"
Jul 29 22:01:10.843449 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.843225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" event={"ID":"e33f0550-085f-49a5-8acf-18c65e367d9b","Type":"ContainerDied","Data":"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"}
Jul 29 22:01:10.843449 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.843281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz" event={"ID":"e33f0550-085f-49a5-8acf-18c65e367d9b","Type":"ContainerDied","Data":"564da87575af66132a7b21c572879e1817a6c7f04acbd1caac4cd4765b5a29f5"}
Jul 29 22:01:10.843449 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.843314    2581 scope.go:117] "RemoveContainer" containerID="929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"
Jul 29 22:01:10.844763 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.844740    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce2567df-882c-43bd-9295-0e16bf260f17" containerID="4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a" exitCode=0
Jul 29 22:01:10.844897 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.844793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"
Jul 29 22:01:10.844897 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.844818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" event={"ID":"ce2567df-882c-43bd-9295-0e16bf260f17","Type":"ContainerDied","Data":"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"}
Jul 29 22:01:10.844897 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.844872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg" event={"ID":"ce2567df-882c-43bd-9295-0e16bf260f17","Type":"ContainerDied","Data":"158bda9b00357af666557dc85e704cfaeafd282fdd86bd2fc8095a6ad3090297"}
Jul 29 22:01:10.869376 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.869347    2581 scope.go:117] "RemoveContainer" containerID="929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"
Jul 29 22:01:10.869851 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:01:10.869816    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d\": container with ID starting with 929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d not found: ID does not exist" containerID="929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"
Jul 29 22:01:10.869961 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.869851    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d"} err="failed to get container status \"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d\": rpc error: code = NotFound desc = could not find container \"929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d\": container with ID starting with 929e69f2db366e31051ab3d299b10b81af5bf2fa33e51efe0efd4dcc4413859d not found: ID does not exist"
Jul 29 22:01:10.869961 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.869871    2581 scope.go:117] "RemoveContainer" containerID="4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"
Jul 29 22:01:10.870227 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.870211    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:01:10.872852 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.872821    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2567df-882c-43bd-9295-0e16bf260f17-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:10.872852 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.872845    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whwg\" (UniqueName: \"kubernetes.io/projected/ce2567df-882c-43bd-9295-0e16bf260f17-kube-api-access-4whwg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:10.872852 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.872856    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33f0550-085f-49a5-8acf-18c65e367d9b-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:10.873082 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.872865    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qtq\" (UniqueName: \"kubernetes.io/projected/e33f0550-085f-49a5-8acf-18c65e367d9b-kube-api-access-28qtq\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:10.873265 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.873243    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-215280-667c8588b8-dxrnz"]
Jul 29 22:01:10.879983 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.879963    2581 scope.go:117] "RemoveContainer" containerID="4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"
Jul 29 22:01:10.880315 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:01:10.880294    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a\": container with ID starting with 4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a not found: ID does not exist" containerID="4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"
Jul 29 22:01:10.880383 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.880325    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a"} err="failed to get container status \"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a\": rpc error: code = NotFound desc = could not find container \"4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a\": container with ID starting with 4381e36f7fa833bc8ff4adfc2bef07a35e699706b4568b1d7fb48bb9afdc8e4a not found: ID does not exist"
Jul 29 22:01:10.884731 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.884683    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:01:10.889936 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:10.889912    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-215280-558ffcc8dd-bfmjg"]
Jul 29 22:01:11.187560 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:11.187524    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2567df-882c-43bd-9295-0e16bf260f17" path="/var/lib/kubelet/pods/ce2567df-882c-43bd-9295-0e16bf260f17/volumes"
Jul 29 22:01:11.187903 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:11.187889    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33f0550-085f-49a5-8acf-18c65e367d9b" path="/var/lib/kubelet/pods/e33f0550-085f-49a5-8acf-18c65e367d9b/volumes"
Jul 29 22:01:36.505193 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.505155    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 22:01:36.505743 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.505377    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" podUID="6d10d950-852c-45aa-a18a-6e24899e18dd" containerName="payload-processing" containerID="cri-o://9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e" gracePeriod=30
Jul 29 22:01:36.514867 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.514837    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 22:01:36.515124 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.515093    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" podUID="5510223c-038d-4614-ab98-f8b047a953a8" containerName="payload-pre-processing" containerID="cri-o://09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218" gracePeriod=30
Jul 29 22:01:36.775561 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.775529    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 22:01:36.809999 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.809969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 22:01:36.927517 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.927468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume\") pod \"6d10d950-852c-45aa-a18a-6e24899e18dd\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") "
Jul 29 22:01:36.927517 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.927528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f2c7\" (UniqueName: \"kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7\") pod \"5510223c-038d-4614-ab98-f8b047a953a8\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") "
Jul 29 22:01:36.927851 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.927546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume\") pod \"5510223c-038d-4614-ab98-f8b047a953a8\" (UID: \"5510223c-038d-4614-ab98-f8b047a953a8\") "
Jul 29 22:01:36.927851 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.927581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pglg6\" (UniqueName: \"kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6\") pod \"6d10d950-852c-45aa-a18a-6e24899e18dd\" (UID: \"6d10d950-852c-45aa-a18a-6e24899e18dd\") "
Jul 29 22:01:36.927975 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.927947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d10d950-852c-45aa-a18a-6e24899e18dd" (UID: "6d10d950-852c-45aa-a18a-6e24899e18dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:01:36.928067 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.928039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5510223c-038d-4614-ab98-f8b047a953a8" (UID: "5510223c-038d-4614-ab98-f8b047a953a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:01:36.929881 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.929859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7" (OuterVolumeSpecName: "kube-api-access-4f2c7") pod "5510223c-038d-4614-ab98-f8b047a953a8" (UID: "5510223c-038d-4614-ab98-f8b047a953a8"). InnerVolumeSpecName "kube-api-access-4f2c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:01:36.930026 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:36.930006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6" (OuterVolumeSpecName: "kube-api-access-pglg6") pod "6d10d950-852c-45aa-a18a-6e24899e18dd" (UID: "6d10d950-852c-45aa-a18a-6e24899e18dd"). InnerVolumeSpecName "kube-api-access-pglg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:01:37.006968 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.006931    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d10d950-852c-45aa-a18a-6e24899e18dd" containerID="9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e" exitCode=0
Jul 29 22:01:37.007178 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.006977    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"
Jul 29 22:01:37.007178 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.007021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" event={"ID":"6d10d950-852c-45aa-a18a-6e24899e18dd","Type":"ContainerDied","Data":"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"}
Jul 29 22:01:37.007178 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.007071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr" event={"ID":"6d10d950-852c-45aa-a18a-6e24899e18dd","Type":"ContainerDied","Data":"aa61709405a8afa58e65f309ffaa70866069a90b92c675a464520a7158808ef6"}
Jul 29 22:01:37.007178 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.007093    2581 scope.go:117] "RemoveContainer" containerID="9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"
Jul 29 22:01:37.008795 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.008772    2581 generic.go:356] "Generic (PLEG): container finished" podID="5510223c-038d-4614-ab98-f8b047a953a8" containerID="09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218" exitCode=0
Jul 29 22:01:37.008904 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.008867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" event={"ID":"5510223c-038d-4614-ab98-f8b047a953a8","Type":"ContainerDied","Data":"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"}
Jul 29 22:01:37.008963 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.008916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q" event={"ID":"5510223c-038d-4614-ab98-f8b047a953a8","Type":"ContainerDied","Data":"ccf876caf61c31a5560ca17bc24097f9295acd30aff3d432d4d542a57e6a1646"}
Jul 29 22:01:37.008963 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.008952    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"
Jul 29 22:01:37.018880 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.018753    2581 scope.go:117] "RemoveContainer" containerID="9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"
Jul 29 22:01:37.019110 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:01:37.019085    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e\": container with ID starting with 9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e not found: ID does not exist" containerID="9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"
Jul 29 22:01:37.019190 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.019122    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e"} err="failed to get container status \"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e\": rpc error: code = NotFound desc = could not find container \"9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e\": container with ID starting with 9ac2992dca78f4b22a2970994c4dab73857b17028df59af434d8bfd80b9fe01e not found: ID does not exist"
Jul 29 22:01:37.019190 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.019152    2581 scope.go:117] "RemoveContainer" containerID="09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"
Jul 29 22:01:37.028126 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028110    2581 scope.go:117] "RemoveContainer" containerID="09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"
Jul 29 22:01:37.028445 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:01:37.028418    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218\": container with ID starting with 09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218 not found: ID does not exist" containerID="09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"
Jul 29 22:01:37.028546 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218"} err="failed to get container status \"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218\": rpc error: code = NotFound desc = could not find container \"09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218\": container with ID starting with 09414bc8533d55f9b926701d2dfbf5308f9d5f481e8b9661c4f09762a267f218 not found: ID does not exist"
Jul 29 22:01:37.028772 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028737    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d950-852c-45aa-a18a-6e24899e18dd-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:37.028772 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028761    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2c7\" (UniqueName: \"kubernetes.io/projected/5510223c-038d-4614-ab98-f8b047a953a8-kube-api-access-4f2c7\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:37.028905 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028777    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5510223c-038d-4614-ab98-f8b047a953a8-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:37.028905 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.028791    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglg6\" (UniqueName: \"kubernetes.io/projected/6d10d950-852c-45aa-a18a-6e24899e18dd-kube-api-access-pglg6\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:01:37.032129 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.032097    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 22:01:37.036641 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.036621    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a0dedf-788f8fbc5d-7l4xr"]
Jul 29 22:01:37.047131 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.047108    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 22:01:37.051436 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.051412    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a0dedf-77f6554fdd-jcb9q"]
Jul 29 22:01:37.188794 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.188752    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5510223c-038d-4614-ab98-f8b047a953a8" path="/var/lib/kubelet/pods/5510223c-038d-4614-ab98-f8b047a953a8/volumes"
Jul 29 22:01:37.189192 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:01:37.189177    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d10d950-852c-45aa-a18a-6e24899e18dd" path="/var/lib/kubelet/pods/6d10d950-852c-45aa-a18a-6e24899e18dd/volumes"
Jul 29 22:02:44.570249 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.570207    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 22:02:44.570899 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.570848    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" podUID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" containerName="payload-processing" containerID="cri-o://c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99" gracePeriod=30
Jul 29 22:02:44.577612 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.577582    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 22:02:44.577855 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.577830    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" podUID="7157533d-7192-442c-9d85-5c5f0e20d2af" containerName="payload-pre-processing" containerID="cri-o://e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1" gracePeriod=30
Jul 29 22:02:44.832140 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.832077    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 22:02:44.863337 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.863307    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 22:02:44.971890 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.971855    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddbvg\" (UniqueName: \"kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg\") pod \"7157533d-7192-442c-9d85-5c5f0e20d2af\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") "
Jul 29 22:02:44.972083 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.971986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn94b\" (UniqueName: \"kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b\") pod \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") "
Jul 29 22:02:44.972083 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.972040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume\") pod \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\" (UID: \"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89\") "
Jul 29 22:02:44.972083 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.972065    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume\") pod \"7157533d-7192-442c-9d85-5c5f0e20d2af\" (UID: \"7157533d-7192-442c-9d85-5c5f0e20d2af\") "
Jul 29 22:02:44.972537 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.972497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" (UID: "de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:02:44.972537 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.972505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7157533d-7192-442c-9d85-5c5f0e20d2af" (UID: "7157533d-7192-442c-9d85-5c5f0e20d2af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:02:44.974176 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.974155    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg" (OuterVolumeSpecName: "kube-api-access-ddbvg") pod "7157533d-7192-442c-9d85-5c5f0e20d2af" (UID: "7157533d-7192-442c-9d85-5c5f0e20d2af"). InnerVolumeSpecName "kube-api-access-ddbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:02:44.974264 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:44.974238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b" (OuterVolumeSpecName: "kube-api-access-hn94b") pod "de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" (UID: "de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89"). InnerVolumeSpecName "kube-api-access-hn94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:02:45.073014 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.072979    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn94b\" (UniqueName: \"kubernetes.io/projected/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-kube-api-access-hn94b\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:02:45.073014 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.073010    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:02:45.073014 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.073019    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7157533d-7192-442c-9d85-5c5f0e20d2af-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:02:45.073327 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.073030    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbvg\" (UniqueName: \"kubernetes.io/projected/7157533d-7192-442c-9d85-5c5f0e20d2af-kube-api-access-ddbvg\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:02:45.423836 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.423798    2581 generic.go:356] "Generic (PLEG): container finished" podID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" containerID="c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99" exitCode=0
Jul 29 22:02:45.424033 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.423849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"
Jul 29 22:02:45.424033 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.423880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" event={"ID":"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89","Type":"ContainerDied","Data":"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"}
Jul 29 22:02:45.424033 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.423932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88" event={"ID":"de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89","Type":"ContainerDied","Data":"62cc89464b912bbb47bfa2e74a6e02d8abf41a4ade9ed599bed38a6edde9d9df"}
Jul 29 22:02:45.424033 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.423965    2581 scope.go:117] "RemoveContainer" containerID="c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"
Jul 29 22:02:45.425411 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.425391    2581 generic.go:356] "Generic (PLEG): container finished" podID="7157533d-7192-442c-9d85-5c5f0e20d2af" containerID="e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1" exitCode=0
Jul 29 22:02:45.425507 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.425446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" event={"ID":"7157533d-7192-442c-9d85-5c5f0e20d2af","Type":"ContainerDied","Data":"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"}
Jul 29 22:02:45.425507 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.425456    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"
Jul 29 22:02:45.425507 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.425483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z" event={"ID":"7157533d-7192-442c-9d85-5c5f0e20d2af","Type":"ContainerDied","Data":"a6081385af0424a7fb0ee201de788fb1fea9e22499fffb1dc3bb2f31b8fedac8"}
Jul 29 22:02:45.433471 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.433453    2581 scope.go:117] "RemoveContainer" containerID="c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"
Jul 29 22:02:45.433793 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:02:45.433760    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99\": container with ID starting with c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99 not found: ID does not exist" containerID="c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"
Jul 29 22:02:45.433897 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.433804    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99"} err="failed to get container status \"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99\": rpc error: code = NotFound desc = could not find container \"c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99\": container with ID starting with c38e1c08d72e707c11350ff67618345c485cb9520385a2ccb5d144732214bd99 not found: ID does not exist"
Jul 29 22:02:45.433897 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.433829    2581 scope.go:117] "RemoveContainer" containerID="e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"
Jul 29 22:02:45.443711 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.443665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 22:02:45.443939 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.443917    2581 scope.go:117] "RemoveContainer" containerID="e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"
Jul 29 22:02:45.444236 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:02:45.444212    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1\": container with ID starting with e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1 not found: ID does not exist" containerID="e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"
Jul 29 22:02:45.444359 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.444239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1"} err="failed to get container status \"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1\": rpc error: code = NotFound desc = could not find container \"e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1\": container with ID starting with e709c189be53425c3d2c98fccf573d129ed7fe115ce4b481f553e344b63581c1 not found: ID does not exist"
Jul 29 22:02:45.447438 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.447372    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50d50c-6c47fbbd58-n8x88"]
Jul 29 22:02:45.473799 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.473763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 22:02:45.482783 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:45.482756    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50d50c-ccf56b66d-nx92z"]
Jul 29 22:02:47.188032 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:47.188003    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7157533d-7192-442c-9d85-5c5f0e20d2af" path="/var/lib/kubelet/pods/7157533d-7192-442c-9d85-5c5f0e20d2af/volumes"
Jul 29 22:02:47.188413 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:02:47.188353    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" path="/var/lib/kubelet/pods/de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89/volumes"
Jul 29 22:03:02.032538 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.032495    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 22:03:02.033155 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.032812    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerName="maas-api" containerID="cri-o://278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260" gracePeriod=30
Jul 29 22:03:02.096150 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.096112    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 22:03:02.096424 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.096379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" podUID="2456de57-c341-479d-bb63-3e9265abb921" containerName="payload-processing" containerID="cri-o://e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6" gracePeriod=30
Jul 29 22:03:02.107424 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.107382    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 22:03:02.107730 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.107662    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" podUID="8ce36441-292b-49a8-bd6e-61df45d81839" containerName="payload-pre-processing" containerID="cri-o://26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138" gracePeriod=30
Jul 29 22:03:02.226575 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.226529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.88:8443/health\": dial tcp 10.133.0.88:8443: connect: connection refused"
Jul 29 22:03:02.392511 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.392482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 22:03:02.411363 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.411317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 22:03:02.426632 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.426599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 22:03:02.526983 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.526942    2581 generic.go:356] "Generic (PLEG): container finished" podID="2456de57-c341-479d-bb63-3e9265abb921" containerID="e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6" exitCode=0
Jul 29 22:03:02.526983 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.526991    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"
Jul 29 22:03:02.527249 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.527030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" event={"ID":"2456de57-c341-479d-bb63-3e9265abb921","Type":"ContainerDied","Data":"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"}
Jul 29 22:03:02.527249 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.527095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k" event={"ID":"2456de57-c341-479d-bb63-3e9265abb921","Type":"ContainerDied","Data":"ca08352be0bc6d59c4408e5c40a30e7de2efb7b3b986256e4842acfbfd4e0ca7"}
Jul 29 22:03:02.527249 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.527123    2581 scope.go:117] "RemoveContainer" containerID="e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"
Jul 29 22:03:02.528425 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.528393    2581 generic.go:356] "Generic (PLEG): container finished" podID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerID="278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260" exitCode=0
Jul 29 22:03:02.528498 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.528432    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"
Jul 29 22:03:02.528498 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.528476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" event={"ID":"11b01367-ab1f-4cd7-a695-7cf463cd2e30","Type":"ContainerDied","Data":"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"}
Jul 29 22:03:02.528584 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.528525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v" event={"ID":"11b01367-ab1f-4cd7-a695-7cf463cd2e30","Type":"ContainerDied","Data":"b454d9528c8a0463e44f6809e090e1ae7abcbf400e2eddf21687fd39e3968102"}
Jul 29 22:03:02.529845 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.529823    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ce36441-292b-49a8-bd6e-61df45d81839" containerID="26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138" exitCode=0
Jul 29 22:03:02.529956 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.529874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"
Jul 29 22:03:02.529956 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.529887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" event={"ID":"8ce36441-292b-49a8-bd6e-61df45d81839","Type":"ContainerDied","Data":"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"}
Jul 29 22:03:02.529956 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.529920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g" event={"ID":"8ce36441-292b-49a8-bd6e-61df45d81839","Type":"ContainerDied","Data":"8e7b680629acb7907cc6436504eae34978696c375aa3971dc780669e840b3a6f"}
Jul 29 22:03:02.536958 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.536934    2581 scope.go:117] "RemoveContainer" containerID="e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"
Jul 29 22:03:02.537264 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:02.537243    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6\": container with ID starting with e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6 not found: ID does not exist" containerID="e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"
Jul 29 22:03:02.537352 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.537275    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6"} err="failed to get container status \"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6\": rpc error: code = NotFound desc = could not find container \"e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6\": container with ID starting with e3a6a46926aa62c6b4a0647faad824b7b617f8a9b17d7c0886534a7ac09ca0d6 not found: ID does not exist"
Jul 29 22:03:02.537352 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.537300    2581 scope.go:117] "RemoveContainer" containerID="278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"
Jul 29 22:03:02.546097 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.546075    2581 scope.go:117] "RemoveContainer" containerID="278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"
Jul 29 22:03:02.546444 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:02.546422    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260\": container with ID starting with 278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260 not found: ID does not exist" containerID="278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"
Jul 29 22:03:02.546524 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.546455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260"} err="failed to get container status \"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260\": rpc error: code = NotFound desc = could not find container \"278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260\": container with ID starting with 278bb361d80792f3756d4d9ec36fd8ce10d4ada7aaa227750e7204f7f5849260 not found: ID does not exist"
Jul 29 22:03:02.546524 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.546477    2581 scope.go:117] "RemoveContainer" containerID="26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"
Jul 29 22:03:02.549858 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.549831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh8fq\" (UniqueName: \"kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq\") pod \"2456de57-c341-479d-bb63-3e9265abb921\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") "
Jul 29 22:03:02.549995 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.549878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume\") pod \"2456de57-c341-479d-bb63-3e9265abb921\" (UID: \"2456de57-c341-479d-bb63-3e9265abb921\") "
Jul 29 22:03:02.549995 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.549918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6j57\" (UniqueName: \"kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57\") pod \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") "
Jul 29 22:03:02.549995 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.549944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume\") pod \"8ce36441-292b-49a8-bd6e-61df45d81839\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") "
Jul 29 22:03:02.549995 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.549975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdglk\" (UniqueName: \"kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk\") pod \"8ce36441-292b-49a8-bd6e-61df45d81839\" (UID: \"8ce36441-292b-49a8-bd6e-61df45d81839\") "
Jul 29 22:03:02.550193 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.550019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca\") pod \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") "
Jul 29 22:03:02.550193 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.550164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") pod \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\" (UID: \"11b01367-ab1f-4cd7-a695-7cf463cd2e30\") "
Jul 29 22:03:02.550435 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.550410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2456de57-c341-479d-bb63-3e9265abb921" (UID: "2456de57-c341-479d-bb63-3e9265abb921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:02.550778 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.550735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ce36441-292b-49a8-bd6e-61df45d81839" (UID: "8ce36441-292b-49a8-bd6e-61df45d81839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:02.550919 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.550448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11b01367-ab1f-4cd7-a695-7cf463cd2e30" (UID: "11b01367-ab1f-4cd7-a695-7cf463cd2e30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:02.552347 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.552322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq" (OuterVolumeSpecName: "kube-api-access-sh8fq") pod "2456de57-c341-479d-bb63-3e9265abb921" (UID: "2456de57-c341-479d-bb63-3e9265abb921"). InnerVolumeSpecName "kube-api-access-sh8fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:02.552462 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.552448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57" (OuterVolumeSpecName: "kube-api-access-x6j57") pod "11b01367-ab1f-4cd7-a695-7cf463cd2e30" (UID: "11b01367-ab1f-4cd7-a695-7cf463cd2e30"). InnerVolumeSpecName "kube-api-access-x6j57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:02.552941 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.552920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11b01367-ab1f-4cd7-a695-7cf463cd2e30" (UID: "11b01367-ab1f-4cd7-a695-7cf463cd2e30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:02.553052 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.553030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk" (OuterVolumeSpecName: "kube-api-access-tdglk") pod "8ce36441-292b-49a8-bd6e-61df45d81839" (UID: "8ce36441-292b-49a8-bd6e-61df45d81839"). InnerVolumeSpecName "kube-api-access-tdglk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:02.556130 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.556099    2581 scope.go:117] "RemoveContainer" containerID="26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"
Jul 29 22:03:02.556491 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:02.556468    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138\": container with ID starting with 26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138 not found: ID does not exist" containerID="26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"
Jul 29 22:03:02.556559 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.556500    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138"} err="failed to get container status \"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138\": rpc error: code = NotFound desc = could not find container \"26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138\": container with ID starting with 26e9069695a9ae14cefa8dcc7b25d83db5c4142b4798034c665ffddd64f3c138 not found: ID does not exist"
Jul 29 22:03:02.651827 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651786    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b01367-ab1f-4cd7-a695-7cf463cd2e30-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.651827 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651820    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8fq\" (UniqueName: \"kubernetes.io/projected/2456de57-c341-479d-bb63-3e9265abb921-kube-api-access-sh8fq\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.651827 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651831    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2456de57-c341-479d-bb63-3e9265abb921-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.652082 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651840    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6j57\" (UniqueName: \"kubernetes.io/projected/11b01367-ab1f-4cd7-a695-7cf463cd2e30-kube-api-access-x6j57\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.652082 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651850    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce36441-292b-49a8-bd6e-61df45d81839-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.652082 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651858    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdglk\" (UniqueName: \"kubernetes.io/projected/8ce36441-292b-49a8-bd6e-61df45d81839-kube-api-access-tdglk\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.652082 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.651867    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b01367-ab1f-4cd7-a695-7cf463cd2e30-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:02.855423 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.855386    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 22:03:02.859084 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.859055    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fe32bc-cc7ffbc64-zbn5g"]
Jul 29 22:03:02.870003 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.869972    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 22:03:02.874400 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.874371    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fe32bc-775cf8bb8b-fp27k"]
Jul 29 22:03:02.886483 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.886452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 22:03:02.891725 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:02.891679    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fe32bc-798889676f-rcv9v"]
Jul 29 22:03:03.188951 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:03.188918    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" path="/var/lib/kubelet/pods/11b01367-ab1f-4cd7-a695-7cf463cd2e30/volumes"
Jul 29 22:03:03.189330 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:03.189285    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2456de57-c341-479d-bb63-3e9265abb921" path="/var/lib/kubelet/pods/2456de57-c341-479d-bb63-3e9265abb921/volumes"
Jul 29 22:03:03.189622 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:03.189608    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce36441-292b-49a8-bd6e-61df45d81839" path="/var/lib/kubelet/pods/8ce36441-292b-49a8-bd6e-61df45d81839/volumes"
Jul 29 22:03:19.752978 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.752940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 22:03:19.753564 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.753203    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" podUID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" containerName="maas-api" containerID="cri-o://f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f" gracePeriod=30
Jul 29 22:03:19.813885 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.813838    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 22:03:19.814221 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.814180    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" podUID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" containerName="payload-processing" containerID="cri-o://10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20" gracePeriod=30
Jul 29 22:03:19.823274 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.823229    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 22:03:19.824912 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:19.824859    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" podUID="b221664d-4e3a-4841-a563-667856555b38" containerName="payload-pre-processing" containerID="cri-o://ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709" gracePeriod=30
Jul 29 22:03:20.088427 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.088340    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 22:03:20.129529 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.129500    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 22:03:20.144242 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.144214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 22:03:20.215133 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215095    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca\") pod \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") "
Jul 29 22:03:20.215340 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") pod \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") "
Jul 29 22:03:20.215340 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb22h\" (UniqueName: \"kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h\") pod \"b221664d-4e3a-4841-a563-667856555b38\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") "
Jul 29 22:03:20.215340 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume\") pod \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") "
Jul 29 22:03:20.215524 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6z26\" (UniqueName: \"kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26\") pod \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\" (UID: \"b9814694-0fe7-4523-a78b-3bf596a6c4aa\") "
Jul 29 22:03:20.215524 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume\") pod \"b221664d-4e3a-4841-a563-667856555b38\" (UID: \"b221664d-4e3a-4841-a563-667856555b38\") "
Jul 29 22:03:20.215524 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215447    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6xvv\" (UniqueName: \"kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv\") pod \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\" (UID: \"07f39ccf-7cde-4f5b-a14f-8529ad841f61\") "
Jul 29 22:03:20.215679 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07f39ccf-7cde-4f5b-a14f-8529ad841f61" (UID: "07f39ccf-7cde-4f5b-a14f-8529ad841f61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:20.215771 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215691    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f39ccf-7cde-4f5b-a14f-8529ad841f61-openshift-service-ca\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.215837 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9814694-0fe7-4523-a78b-3bf596a6c4aa" (UID: "b9814694-0fe7-4523-a78b-3bf596a6c4aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:20.216013 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.215913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b221664d-4e3a-4841-a563-667856555b38" (UID: "b221664d-4e3a-4841-a563-667856555b38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:20.217620 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.217601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07f39ccf-7cde-4f5b-a14f-8529ad841f61" (UID: "07f39ccf-7cde-4f5b-a14f-8529ad841f61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:20.217804 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.217636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26" (OuterVolumeSpecName: "kube-api-access-m6z26") pod "b9814694-0fe7-4523-a78b-3bf596a6c4aa" (UID: "b9814694-0fe7-4523-a78b-3bf596a6c4aa"). InnerVolumeSpecName "kube-api-access-m6z26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:20.217804 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.217644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv" (OuterVolumeSpecName: "kube-api-access-w6xvv") pod "07f39ccf-7cde-4f5b-a14f-8529ad841f61" (UID: "07f39ccf-7cde-4f5b-a14f-8529ad841f61"). InnerVolumeSpecName "kube-api-access-w6xvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:20.217804 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.217673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h" (OuterVolumeSpecName: "kube-api-access-jb22h") pod "b221664d-4e3a-4841-a563-667856555b38" (UID: "b221664d-4e3a-4841-a563-667856555b38"). InnerVolumeSpecName "kube-api-access-jb22h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316890    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6z26\" (UniqueName: \"kubernetes.io/projected/b9814694-0fe7-4523-a78b-3bf596a6c4aa-kube-api-access-m6z26\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316919    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b221664d-4e3a-4841-a563-667856555b38-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316932    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6xvv\" (UniqueName: \"kubernetes.io/projected/07f39ccf-7cde-4f5b-a14f-8529ad841f61-kube-api-access-w6xvv\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316945    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f39ccf-7cde-4f5b-a14f-8529ad841f61-maas-api-tls\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316955    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb22h\" (UniqueName: \"kubernetes.io/projected/b221664d-4e3a-4841-a563-667856555b38-kube-api-access-jb22h\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.316976 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.316964    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9814694-0fe7-4523-a78b-3bf596a6c4aa-plugins-config-volume\") on node \"ip-10-0-143-168.ec2.internal\" DevicePath \"\""
Jul 29 22:03:20.638998 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.638962    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" containerID="10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20" exitCode=0
Jul 29 22:03:20.639187 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.639015    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"
Jul 29 22:03:20.639187 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.639054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" event={"ID":"b9814694-0fe7-4523-a78b-3bf596a6c4aa","Type":"ContainerDied","Data":"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"}
Jul 29 22:03:20.639187 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.639115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5" event={"ID":"b9814694-0fe7-4523-a78b-3bf596a6c4aa","Type":"ContainerDied","Data":"b0af7c798f4eb573c2ca85252ea2309da951a7c958f569d60f118dbe4af1cc03"}
Jul 29 22:03:20.639187 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.639153    2581 scope.go:117] "RemoveContainer" containerID="10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"
Jul 29 22:03:20.640561 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.640537    2581 generic.go:356] "Generic (PLEG): container finished" podID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" containerID="f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f" exitCode=0
Jul 29 22:03:20.640675 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.640593    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"
Jul 29 22:03:20.640770 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.640602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" event={"ID":"07f39ccf-7cde-4f5b-a14f-8529ad841f61","Type":"ContainerDied","Data":"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"}
Jul 29 22:03:20.640822 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.640809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44" event={"ID":"07f39ccf-7cde-4f5b-a14f-8529ad841f61","Type":"ContainerDied","Data":"43af537e80c9f2f27d522b086dc9cf25bc0e78e397f0b9b909a1d8eb0d588f9c"}
Jul 29 22:03:20.642095 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.642075    2581 generic.go:356] "Generic (PLEG): container finished" podID="b221664d-4e3a-4841-a563-667856555b38" containerID="ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709" exitCode=0
Jul 29 22:03:20.642227 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.642103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" event={"ID":"b221664d-4e3a-4841-a563-667856555b38","Type":"ContainerDied","Data":"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"}
Jul 29 22:03:20.642227 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.642117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"
Jul 29 22:03:20.642227 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.642137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh" event={"ID":"b221664d-4e3a-4841-a563-667856555b38","Type":"ContainerDied","Data":"5a2e3f976bb642a571065feca4365041e9fce5c8c044d76110019b5319a4f1c1"}
Jul 29 22:03:20.650194 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.650170    2581 scope.go:117] "RemoveContainer" containerID="10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"
Jul 29 22:03:20.650828 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:20.650560    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20\": container with ID starting with 10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20 not found: ID does not exist" containerID="10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"
Jul 29 22:03:20.650828 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.650601    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20"} err="failed to get container status \"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20\": rpc error: code = NotFound desc = could not find container \"10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20\": container with ID starting with 10da86e0ea7fb3a161ad28fb2721accd9ba447d39b8148a79c21e281b922be20 not found: ID does not exist"
Jul 29 22:03:20.650828 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.650625    2581 scope.go:117] "RemoveContainer" containerID="f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"
Jul 29 22:03:20.660346 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.660322    2581 scope.go:117] "RemoveContainer" containerID="f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"
Jul 29 22:03:20.660662 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:20.660642    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f\": container with ID starting with f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f not found: ID does not exist" containerID="f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"
Jul 29 22:03:20.660778 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.660674    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f"} err="failed to get container status \"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f\": rpc error: code = NotFound desc = could not find container \"f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f\": container with ID starting with f3f237c313f2c1f40d142b29da6f04464632512903296f8ee926f95319666e7f not found: ID does not exist"
Jul 29 22:03:20.660778 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.660715    2581 scope.go:117] "RemoveContainer" containerID="ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"
Jul 29 22:03:20.670314 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.670283    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 22:03:20.671599 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.671541    2581 scope.go:117] "RemoveContainer" containerID="ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"
Jul 29 22:03:20.672005 ip-10-0-143-168 kubenswrapper[2581]: E0729 22:03:20.671978    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709\": container with ID starting with ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709 not found: ID does not exist" containerID="ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"
Jul 29 22:03:20.672104 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.672015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709"} err="failed to get container status \"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709\": rpc error: code = NotFound desc = could not find container \"ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709\": container with ID starting with ae99b9826ce049c28ddc41adf1c36fc9a2f94bb12e9b39d911996783c8750709 not found: ID does not exist"
Jul 29 22:03:20.674497 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.674474    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-670ddc-865dddffdb-lqrf5"]
Jul 29 22:03:20.688837 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.688809    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 22:03:20.692463 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.692430    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-670ddc-6fb95c8d99-d9f44"]
Jul 29 22:03:20.705231 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.705201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 22:03:20.715478 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:20.715449    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-670ddc-655b8f446d-8xfkh"]
Jul 29 22:03:21.187339 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:21.187304    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" path="/var/lib/kubelet/pods/07f39ccf-7cde-4f5b-a14f-8529ad841f61/volumes"
Jul 29 22:03:21.187789 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:21.187774    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b221664d-4e3a-4841-a563-667856555b38" path="/var/lib/kubelet/pods/b221664d-4e3a-4841-a563-667856555b38/volumes"
Jul 29 22:03:21.188163 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:21.188147    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" path="/var/lib/kubelet/pods/b9814694-0fe7-4523-a78b-3bf596a6c4aa/volumes"
Jul 29 22:03:27.483585 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:27.483491    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c986c7ccc-w2cxs_6b12b766-ac8c-489f-8910-acaedac3a00e/authorino/0.log"
Jul 29 22:03:33.423411 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.423335    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6469f67b9b-bw85b_eb72052e-be0a-453e-b486-393662938058/manager/0.log"
Jul 29 22:03:33.486195 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.486073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 22:03:33.503999 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.501264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2xwnk_3f40bd58-fd66-415b-9dc8-a7235a1e1316/ovn-acl-logging/0.log"
Jul 29 22:03:33.552329 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.552293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zzq88_f98a9db3-fe2d-4568-b811-aa1d76890856/server/0.log"
Jul 29 22:03:33.672933 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.672901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mxbnj_f9fe751a-da54-4097-8dea-d483673e6a62/manager/0.log"
Jul 29 22:03:33.933913 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.933880    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446ffcd9c-cr54w_1440618b-50cd-4d74-a105-c30046991ae0/copy-manifests/0.log"
Jul 29 22:03:33.947898 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:33.947868    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446ffcd9c-cr54w_1440618b-50cd-4d74-a105-c30046991ae0/manager/0.log"
Jul 29 22:03:34.455586 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:34.455560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b5d9dbbcd-p5vtv_2d0bdf8f-fefb-4f89-ac50-a2973b77fa54/maas-api/0.log"
Jul 29 22:03:34.931426 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:34.931397    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tmk4c_443b3da5-ce8f-41ff-bef9-439e20baef03/postgres/0.log"
Jul 29 22:03:35.770865 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.770828    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/util/0.log"
Jul 29 22:03:35.779055 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.779028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/pull/0.log"
Jul 29 22:03:35.785811 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.785778    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/extract/0.log"
Jul 29 22:03:35.918654 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.918623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/util/0.log"
Jul 29 22:03:35.926480 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.926455    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/pull/0.log"
Jul 29 22:03:35.934660 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:35.934635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/extract/0.log"
Jul 29 22:03:36.057021 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.056941    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/util/0.log"
Jul 29 22:03:36.065425 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.065401    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/pull/0.log"
Jul 29 22:03:36.074234 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.074209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/extract/0.log"
Jul 29 22:03:36.195953 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.195928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/extract/0.log"
Jul 29 22:03:36.202466 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.202439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/util/0.log"
Jul 29 22:03:36.209350 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.209319    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/pull/0.log"
Jul 29 22:03:36.337632 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.337528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c986c7ccc-w2cxs_6b12b766-ac8c-489f-8910-acaedac3a00e/authorino/0.log"
Jul 29 22:03:36.464869 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.464836    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kxvdz_ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25/manager/0.log"
Jul 29 22:03:36.586148 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.586110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8njwz_70072b7d-14d8-4e9b-949a-ce93034a8c7a/manager/0.log"
Jul 29 22:03:36.715333 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.715305    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfkl8_c294ae37-68cc-4277-a641-086991f8f09f/kuadrant-console-plugin/0.log"
Jul 29 22:03:36.838753 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.838722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fqfh4_e89c8789-9fb4-4eac-92c5-cfe167a3a3b2/registry-server/0.log"
Jul 29 22:03:36.965051 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:36.965019    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jsz76_82559be4-d305-4ac1-93a8-fe85fc07a4ca/manager/0.log"
Jul 29 22:03:37.217839 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:37.217796    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m5szc_30b70b50-e020-47cc-b1e0-c00cc9f167e9/manager/0.log"
Jul 29 22:03:37.985790 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:37.985761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d94d9cdd-4kfsc_11c7faca-78cb-4dcc-bfe1-23b0dbb40376/kube-auth-proxy/0.log"
Jul 29 22:03:38.254944 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:38.254865    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj8hl_71c41b59-4611-4075-81e2-ae896e03ca9d/payload-pre-processing/0.log"
Jul 29 22:03:38.378416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:38.378379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 22:03:38.859297 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:38.859246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xcj6v_fc3043ba-32b5-4c15-bb95-4569c88fff70/main/0.log"
Jul 29 22:03:39.218995 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:39.218910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvwbg_e8f01d9a-d3ee-4278-915f-3f4d6e335d37/main/0.log"
Jul 29 22:03:39.344835 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:39.344805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-g2d4q_6f0dbec1-f992-491e-b6fd-2795caebbeea/main/0.log"
Jul 29 22:03:39.473919 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:39.473836    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-58thp_694f2fcd-0c10-4847-804b-c0d880901186/main/0.log"
Jul 29 22:03:48.618779 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:48.618741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t4q5p_deb2dcc5-86f5-4cb7-be2a-826aa8b04b60/global-pull-secret-syncer/0.log"
Jul 29 22:03:48.712476 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:48.712441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8vmkk_de8b18af-2228-40dc-b006-01407c1c8ba7/konnectivity-agent/0.log"
Jul 29 22:03:48.818123 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:48.818091    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-168.ec2.internal_df72579e9d2f6ec664c78ab551d5adbe/haproxy/0.log"
Jul 29 22:03:52.583591 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.583553    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/extract/0.log"
Jul 29 22:03:52.611946 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.611911    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/util/0.log"
Jul 29 22:03:52.637449 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.637412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594b58x_382fa42b-c64f-4c9d-a6b2-b2c7b929e319/pull/0.log"
Jul 29 22:03:52.694293 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.694246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/extract/0.log"
Jul 29 22:03:52.719579 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.719554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/util/0.log"
Jul 29 22:03:52.747565 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.747540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zlrd4_f7f9a784-1522-4a63-a361-96197dbce877/pull/0.log"
Jul 29 22:03:52.775742 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.775711    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/extract/0.log"
Jul 29 22:03:52.803923 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.803898    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/util/0.log"
Jul 29 22:03:52.823861 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.823837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vv2t8_60c2979e-b10f-4c08-8da5-8cf8444f66ab/pull/0.log"
Jul 29 22:03:52.856511 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.856482    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/extract/0.log"
Jul 29 22:03:52.887656 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.887627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/util/0.log"
Jul 29 22:03:52.916484 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:52.916460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n75sj_c06b0603-f08e-4e5f-b8b3-b2685fe1de50/pull/0.log"
Jul 29 22:03:53.295633 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.295541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c986c7ccc-w2cxs_6b12b766-ac8c-489f-8910-acaedac3a00e/authorino/0.log"
Jul 29 22:03:53.333588 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.333555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kxvdz_ef69bd10-a7ee-49c0-9478-c3ccb8e9ff25/manager/0.log"
Jul 29 22:03:53.363285 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.363261    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8njwz_70072b7d-14d8-4e9b-949a-ce93034a8c7a/manager/0.log"
Jul 29 22:03:53.404640 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.404610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zfkl8_c294ae37-68cc-4277-a641-086991f8f09f/kuadrant-console-plugin/0.log"
Jul 29 22:03:53.459104 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.459067    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fqfh4_e89c8789-9fb4-4eac-92c5-cfe167a3a3b2/registry-server/0.log"
Jul 29 22:03:53.624678 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.624643    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jsz76_82559be4-d305-4ac1-93a8-fe85fc07a4ca/manager/0.log"
Jul 29 22:03:53.735179 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:53.735141    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m5szc_30b70b50-e020-47cc-b1e0-c00cc9f167e9/manager/0.log"
Jul 29 22:03:55.970707 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:55.970677    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2t7f_b007e500-cf5e-428a-82af-9c9348d49d0a/node-exporter/0.log"
Jul 29 22:03:55.996254 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:55.996224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2t7f_b007e500-cf5e-428a-82af-9c9348d49d0a/kube-rbac-proxy/0.log"
Jul 29 22:03:56.023610 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:56.023592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2t7f_b007e500-cf5e-428a-82af-9c9348d49d0a/init-textfile/0.log"
Jul 29 22:03:58.801010 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.800974    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"]
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce36441-292b-49a8-bd6e-61df45d81839" containerName="payload-pre-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801338    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce36441-292b-49a8-bd6e-61df45d81839" containerName="payload-pre-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801365    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerName="maas-api"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801371    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerName="maas-api"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801381    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" containerName="payload-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" containerName="payload-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801395    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7157533d-7192-442c-9d85-5c5f0e20d2af" containerName="payload-pre-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801401    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7157533d-7192-442c-9d85-5c5f0e20d2af" containerName="payload-pre-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801406    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33f0550-085f-49a5-8acf-18c65e367d9b" containerName="payload-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801412    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33f0550-085f-49a5-8acf-18c65e367d9b" containerName="payload-processing"
Jul 29 22:03:58.801416 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801419    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d10d950-852c-45aa-a18a-6e24899e18dd" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d10d950-852c-45aa-a18a-6e24899e18dd" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801436    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2456de57-c341-479d-bb63-3e9265abb921" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801441    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2456de57-c341-479d-bb63-3e9265abb921" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801448    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" containerName="maas-api"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" containerName="maas-api"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801458    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5510223c-038d-4614-ab98-f8b047a953a8" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801463    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5510223c-038d-4614-ab98-f8b047a953a8" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b221664d-4e3a-4841-a563-667856555b38" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b221664d-4e3a-4841-a563-667856555b38" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2567df-882c-43bd-9295-0e16bf260f17" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801512    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2567df-882c-43bd-9295-0e16bf260f17" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801560    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b01367-ab1f-4cd7-a695-7cf463cd2e30" containerName="maas-api"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801566    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9814694-0fe7-4523-a78b-3bf596a6c4aa" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801571    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0d79b3-c8a2-4768-a6ed-2cdbc2a53d89" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33f0550-085f-49a5-8acf-18c65e367d9b" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801582    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7157533d-7192-442c-9d85-5c5f0e20d2af" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801588    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b221664d-4e3a-4841-a563-667856555b38" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801593    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f39ccf-7cde-4f5b-a14f-8529ad841f61" containerName="maas-api"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801598    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d10d950-852c-45aa-a18a-6e24899e18dd" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801604    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2567df-882c-43bd-9295-0e16bf260f17" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801609    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2456de57-c341-479d-bb63-3e9265abb921" containerName="payload-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801614    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce36441-292b-49a8-bd6e-61df45d81839" containerName="payload-pre-processing"
Jul 29 22:03:58.801782 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.801620    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5510223c-038d-4614-ab98-f8b047a953a8" containerName="payload-pre-processing"
Jul 29 22:03:58.805828 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.805793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.812019 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.811964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xq8zq\"/\"default-dockercfg-9jqsr\""
Jul 29 22:03:58.812312 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.812287    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xq8zq\"/\"kube-root-ca.crt\""
Jul 29 22:03:58.812431 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.812340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xq8zq\"/\"openshift-service-ca.crt\""
Jul 29 22:03:58.815154 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.814845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"]
Jul 29 22:03:58.858653 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.858621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6qj\" (UniqueName: \"kubernetes.io/projected/da8edd1a-52da-4fdc-8040-5972074035da-kube-api-access-xf6qj\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.858887 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.858850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-proc\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.858965 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.858907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-lib-modules\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.859021 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.858976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-sys\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.859075 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.859041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-podres\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960145 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-proc\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-lib-modules\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-sys\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-proc\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-podres\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-sys\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6qj\" (UniqueName: \"kubernetes.io/projected/da8edd1a-52da-4fdc-8040-5972074035da-kube-api-access-xf6qj\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-podres\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.960366 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.960350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da8edd1a-52da-4fdc-8040-5972074035da-lib-modules\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:58.970413 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:58.970383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6qj\" (UniqueName: \"kubernetes.io/projected/da8edd1a-52da-4fdc-8040-5972074035da-kube-api-access-xf6qj\") pod \"perf-node-gather-daemonset-6csxt\" (UID: \"da8edd1a-52da-4fdc-8040-5972074035da\") " pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:59.119793 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.119761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:59.248928 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.248897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d94c59877-9vf9k_4c2cc28a-8269-4fd7-9a46-004bfb3a4835/console/0.log"
Jul 29 22:03:59.290760 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.290728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"]
Jul 29 22:03:59.292738 ip-10-0-143-168 kubenswrapper[2581]: W0729 22:03:59.292691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda8edd1a_52da_4fdc_8040_5972074035da.slice/crio-e733210d5c334740477cc27e7a81e8faf082381bcb682e13f0fb37a2d9dd3a8a WatchSource:0}: Error finding container e733210d5c334740477cc27e7a81e8faf082381bcb682e13f0fb37a2d9dd3a8a: Status 404 returned error can't find the container with id e733210d5c334740477cc27e7a81e8faf082381bcb682e13f0fb37a2d9dd3a8a
Jul 29 22:03:59.873684 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.873646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt" event={"ID":"da8edd1a-52da-4fdc-8040-5972074035da","Type":"ContainerStarted","Data":"55e450d974990c20e658cb3eb0d09a0ec03743450122e2f867def2bdaf5f679d"}
Jul 29 22:03:59.874097 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.873722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt" event={"ID":"da8edd1a-52da-4fdc-8040-5972074035da","Type":"ContainerStarted","Data":"e733210d5c334740477cc27e7a81e8faf082381bcb682e13f0fb37a2d9dd3a8a"}
Jul 29 22:03:59.874097 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.873751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:03:59.891676 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:03:59.891618    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt" podStartSLOduration=1.891603626 podStartE2EDuration="1.891603626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:59.888666418 +0000 UTC m=+3627.187144174" watchObservedRunningTime="2026-07-29 22:03:59.891603626 +0000 UTC m=+3627.190081380"
Jul 29 22:04:01.034177 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:01.034141    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8hsh_72d5e612-cb4c-4305-bfbf-931fa65818c7/dns/0.log"
Jul 29 22:04:01.055902 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:01.055854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8hsh_72d5e612-cb4c-4305-bfbf-931fa65818c7/kube-rbac-proxy/0.log"
Jul 29 22:04:01.153955 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:01.153922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dn9wv_a07af5c6-3d90-4231-9f57-63f3566f8f62/dns-node-resolver/0.log"
Jul 29 22:04:01.718789 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:01.718761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b6b796dcf-znxdx_80076396-c68c-474f-b03f-5de5cc07cd87/registry/0.log"
Jul 29 22:04:01.787637 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:01.787602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t9785_e2a2e0e9-8711-4116-a953-cfd4d2b5c978/node-ca/0.log"
Jul 29 22:04:02.914301 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:02.914266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d94d9cdd-4kfsc_11c7faca-78cb-4dcc-bfe1-23b0dbb40376/kube-auth-proxy/0.log"
Jul 29 22:04:03.014743 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:03.014691    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj8hl_71c41b59-4611-4075-81e2-ae896e03ca9d/payload-pre-processing/0.log"
Jul 29 22:04:03.045990 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:03.045958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 22:04:03.967686 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:03.967654    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jzv4d_b32cd7f0-fea6-4130-a158-a141dfff3207/serve-healthcheck-canary/0.log"
Jul 29 22:04:04.467245 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:04.467212    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wqz9_4d2c23cf-45d3-40d4-97e8-4e906e60e4db/kube-rbac-proxy/0.log"
Jul 29 22:04:04.491228 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:04.491201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wqz9_4d2c23cf-45d3-40d4-97e8-4e906e60e4db/exporter/0.log"
Jul 29 22:04:04.513498 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:04.513473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wqz9_4d2c23cf-45d3-40d4-97e8-4e906e60e4db/extractor/0.log"
Jul 29 22:04:05.890197 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:05.890169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xq8zq/perf-node-gather-daemonset-6csxt"
Jul 29 22:04:07.206262 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:07.206213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6469f67b9b-bw85b_eb72052e-be0a-453e-b486-393662938058/manager/0.log"
Jul 29 22:04:07.251234 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:07.251210    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zzq88_f98a9db3-fe2d-4568-b811-aa1d76890856/server/0.log"
Jul 29 22:04:07.327609 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:07.327576    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mxbnj_f9fe751a-da54-4097-8dea-d483673e6a62/manager/0.log"
Jul 29 22:04:07.476159 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:07.476129    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446ffcd9c-cr54w_1440618b-50cd-4d74-a105-c30046991ae0/manager/0.log"
Jul 29 22:04:07.493803 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:07.493772    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446ffcd9c-cr54w_1440618b-50cd-4d74-a105-c30046991ae0/copy-manifests/0.log"
Jul 29 22:04:09.656907 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:09.656884    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d94d9cdd-4kfsc_11c7faca-78cb-4dcc-bfe1-23b0dbb40376/kube-auth-proxy/0.log"
Jul 29 22:04:09.741368 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:09.741322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj8hl_71c41b59-4611-4075-81e2-ae896e03ca9d/payload-pre-processing/0.log"
Jul 29 22:04:09.766741 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:09.766706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 22:04:11.319021 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:11.318985    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75d94d9cdd-4kfsc_11c7faca-78cb-4dcc-bfe1-23b0dbb40376/kube-auth-proxy/0.log"
Jul 29 22:04:11.394064 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:11.394031    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj8hl_71c41b59-4611-4075-81e2-ae896e03ca9d/payload-pre-processing/0.log"
Jul 29 22:04:11.416904 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:11.416873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-rngfn_9423ba19-ac0d-42e1-8948-77a5aa291ec5/payload-processing/0.log"
Jul 29 22:04:15.463018 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.462990    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/kube-multus-additional-cni-plugins/0.log"
Jul 29 22:04:15.484966 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.484937    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/egress-router-binary-copy/0.log"
Jul 29 22:04:15.506779 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.506752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/cni-plugins/0.log"
Jul 29 22:04:15.527714 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.527676    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/bond-cni-plugin/0.log"
Jul 29 22:04:15.550179 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.550151    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/routeoverride-cni/0.log"
Jul 29 22:04:15.575188 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.575159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/whereabouts-cni-bincopy/0.log"
Jul 29 22:04:15.598521 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.598486    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk8nn_d380a421-1160-4abb-9afb-7bb0adff6d05/whereabouts-cni/0.log"
Jul 29 22:04:15.839739 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.839648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g4xnp_23e94a11-f3fc-4b84-b714-3667c90e23c7/kube-multus/0.log"
Jul 29 22:04:15.899316 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.899289    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tv28_b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2/network-metrics-daemon/0.log"
Jul 29 22:04:15.917374 ip-10-0-143-168 kubenswrapper[2581]: I0729 22:04:15.917335    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4tv28_b669f1f7-e1f3-48ba-8d67-fb7f94bda5c2/kube-rbac-proxy/0.log"